{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Processing data using Spark Dataframe with Pyspark"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "View spark version. If you get an error, you need to troubleshoot the reason further."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'2.3.0'"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "spark.version"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Exercise 1: use movielens dataset for the following exercise\n",
    "\n",
    "1. Load movies.csv as movies dataframe. Cache the dataframe\n",
    "2. Load ratings.csv as ratings dataframe. Cache the dataframe\n",
    "3. Find the number of records in movies dataframe\n",
    "4. Find the number of records in ratings dataframe\n",
    "5. Validate the userId and movieId combination is unique\n",
    "6. Find average rating and count of rating per movieId using ratings dataframe\n",
    "7. Find top 10 movies based on the highest average ratings. Consider only those movies that have at least 100 ratings. Show movieId, title, average rating and rating count columns.\n",
    "8. Show temporary views for current Spark session\n",
    "9. Register movies dataframe and ratings dayaframe as movies and ratings temporary view respectively. Verify that you can see the new temporary views you just created.\n",
    "10. Using SQL statement, solve the problem statement for step #7. Match the results from step #7."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Load Spark SQL functions, for example: count, avg, explode etc."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "from pyspark.sql.functions import *"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Location of movies dataset. You can download the dataset from here. Here we are using latest-small dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "home_dir = \"/user/cloudera/movielens/\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Create a dataframe on movies.csv file"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "movies = (spark.read.format(\"csv\")\n",
    ".options(header = True, inferSchema = True)\n",
    ".load(home_dir + \"movies.csv\")\n",
    ".cache()) # Keep the dataframe in memory for faster processing "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Show schema of movies dataframe"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "root\n",
      " |-- movieId: integer (nullable = true)\n",
      " |-- title: string (nullable = true)\n",
      " |-- genres: string (nullable = true)\n",
      "\n"
     ]
    }
   ],
   "source": [
    "movies.printSchema()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('movieId', 'int'), ('title', 'string'), ('genres', 'string')]"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "movies.dtypes"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Display a few sample view from movies Dataframe"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+-------+--------------------+--------------------+\n",
      "|movieId|               title|              genres|\n",
      "+-------+--------------------+--------------------+\n",
      "|      1|    Toy Story (1995)|Adventure|Animati...|\n",
      "|      2|      Jumanji (1995)|Adventure|Childre...|\n",
      "|      3|Grumpier Old Men ...|      Comedy|Romance|\n",
      "|      4|Waiting to Exhale...|Comedy|Drama|Romance|\n",
      "|      5|Father of the Bri...|              Comedy|\n",
      "+-------+--------------------+--------------------+\n",
      "only showing top 5 rows\n",
      "\n"
     ]
    }
   ],
   "source": [
    "movies.show(5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Find the number of records in movies dataframe"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "10329"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "movies.count()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Create ratings Dataframe using ratings.csv file"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "ratings = (spark.read.format(\"csv\")\n",
    ".options(header = True, inferSchema = True)\n",
    ".load(home_dir + \"ratings.csv\")\n",
    ".persist())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Print schema of ratings Dataframe"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "root\n",
      " |-- userId: integer (nullable = true)\n",
      " |-- movieId: integer (nullable = true)\n",
      " |-- rating: double (nullable = true)\n",
      " |-- timestamp: integer (nullable = true)\n",
      "\n"
     ]
    }
   ],
   "source": [
    "ratings.printSchema()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Show a few sample values from ratings dataframe"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+------+-------+------+----------+\n",
      "|userId|movieId|rating| timestamp|\n",
      "+------+-------+------+----------+\n",
      "|     1|     16|   4.0|1217897793|\n",
      "|     1|     24|   1.5|1217895807|\n",
      "|     1|     32|   4.0|1217896246|\n",
      "|     1|     47|   4.0|1217896556|\n",
      "|     1|     50|   4.0|1217896523|\n",
      "+------+-------+------+----------+\n",
      "only showing top 5 rows\n",
      "\n"
     ]
    }
   ],
   "source": [
    "ratings.show(5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Find the number of records in ratings"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "105339"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ratings.count()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Validate the movieId and ratingId combination is unique identifier in ratings table"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+-------+------+-----+\n",
      "|movieId|userId|count|\n",
      "+-------+------+-----+\n",
      "+-------+------+-----+\n",
      "\n"
     ]
    }
   ],
   "source": [
    "ratings.groupBy(\"movieId\", \"userId\").count().filter(\"count != 1\").show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As it shows that there is no userId and movieId combination that occurs more than once. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Find average rating of each movie for which there are at least 100 ratings. Order the result by average rating in decreasing order."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+-------+-----+------------------+\n",
      "|movieId|count|        avg_rating|\n",
      "+-------+-----+------------------+\n",
      "|   1580|  172| 3.627906976744186|\n",
      "|    471|   48|3.6666666666666665|\n",
      "|   7253|    2|               4.0|\n",
      "|   3997|   11|               2.5|\n",
      "|  32460|    1|               4.5|\n",
      "|  44022|   16|            3.3125|\n",
      "|  54190|    7| 3.357142857142857|\n",
      "|    833|   10|              2.95|\n",
      "|   1591|   33| 2.257575757575758|\n",
      "|   6357|    5|               3.6|\n",
      "|   2366|   31| 3.532258064516129|\n",
      "|   1088|   56| 3.419642857142857|\n",
      "|   1238|   12|              3.75|\n",
      "|   4519|   11|3.0454545454545454|\n",
      "|   1645|   60| 3.566666666666667|\n",
      "|   8638|   18|3.9722222222222223|\n",
      "|   3175|   73| 3.712328767123288|\n",
      "|   2122|   14|2.2142857142857144|\n",
      "|  26755|    1|               0.5|\n",
      "|   5803|    7|2.5714285714285716|\n",
      "+-------+-----+------------------+\n",
      "only showing top 20 rows\n",
      "\n"
     ]
    }
   ],
   "source": [
    "ratings_agg = (ratings\n",
    ".groupBy(col(\"movieId\"))\n",
    ".agg(\n",
    "    count(col(\"movieId\")).alias(\"count\"),\n",
    "    avg(col(\"rating\")).alias(\"avg_rating\")\n",
    "))\n",
    "\n",
    "ratings_agg.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+-------+--------------------+------------------+-----+\n",
      "|movieId|               title|        avg_rating|count|\n",
      "+-------+--------------------+------------------+-----+\n",
      "|    318|Shawshank Redempt...| 4.454545454545454|  308|\n",
      "|    858|Godfather, The (1...| 4.392857142857143|  210|\n",
      "|     50|Usual Suspects, T...| 4.328947368421052|  228|\n",
      "|   1136|Monty Python and ...|4.3019480519480515|  154|\n",
      "|    527|Schindler's List ...| 4.296370967741935|  248|\n",
      "|   1193|One Flew Over the...|4.2727272727272725|  143|\n",
      "|    608|        Fargo (1996)|4.2711442786069655|  201|\n",
      "|   2571|  Matrix, The (1999)| 4.264367816091954|  261|\n",
      "|   1221|Godfather: Part I...| 4.260714285714286|  140|\n",
      "|   1213|   Goodfellas (1990)|4.2592592592592595|  135|\n",
      "+-------+--------------------+------------------+-----+\n",
      "\n"
     ]
    }
   ],
   "source": [
    "(ratings_agg\n",
    ".alias(\"t1\")\n",
    ".join(movies.alias(\"t2\"), col(\"t1.movieId\") == col(\"t2.movieId\"))\n",
    ".filter(\"count > 100\")\n",
    ".orderBy(desc(\"avg_rating\"))\n",
    ".select(\"t1.movieId\", \"title\", \"avg_rating\", \"count\")\n",
    ".limit(10)\n",
    ".show())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Show temporary views for current Spark session"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+--------+---------+-----------+\n",
      "|database|tableName|isTemporary|\n",
      "+--------+---------+-----------+\n",
      "| default|   movies|      false|\n",
      "| default|  weblogs|      false|\n",
      "+--------+---------+-----------+\n",
      "\n"
     ]
    }
   ],
   "source": [
    "sql(\"show tables\").show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+--------+---------+-----------+\n",
      "|database|tableName|isTemporary|\n",
      "+--------+---------+-----------+\n",
      "| default|   movies|      false|\n",
      "| default|  weblogs|      false|\n",
      "|        |   movies|       true|\n",
      "|        |  ratings|       true|\n",
      "+--------+---------+-----------+\n",
      "\n"
     ]
    }
   ],
   "source": [
    "movies.createOrReplaceTempView(\"movies\")\n",
    "ratings.createOrReplaceTempView(\"ratings\")\n",
    "sql(\"show tables\").show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Using SQL statement, find top 10 movies based on the highest average ratings. Consider only those movies that have at least 100 ratings. Show movieId, title, average rating and rating count columns."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+-------+--------------------+------------------+------------+\n",
      "|movieId|               title|        avg_rating|rating_count|\n",
      "+-------+--------------------+------------------+------------+\n",
      "|    318|Shawshank Redempt...| 4.454545454545454|         308|\n",
      "|    858|Godfather, The (1...| 4.392857142857143|         210|\n",
      "|     50|Usual Suspects, T...| 4.328947368421052|         228|\n",
      "|   1136|Monty Python and ...|4.3019480519480515|         154|\n",
      "|    527|Schindler's List ...| 4.296370967741935|         248|\n",
      "|   1193|One Flew Over the...|4.2727272727272725|         143|\n",
      "|    608|        Fargo (1996)|4.2711442786069655|         201|\n",
      "|   2571|  Matrix, The (1999)| 4.264367816091954|         261|\n",
      "|   1221|Godfather: Part I...| 4.260714285714286|         140|\n",
      "|   1213|   Goodfellas (1990)|4.2592592592592595|         135|\n",
      "+-------+--------------------+------------------+------------+\n",
      "\n"
     ]
    }
   ],
   "source": [
    "sql(\"\"\"\n",
    "\n",
    "select \n",
    "    t1.movieId, \n",
    "    t1.title, \n",
    "    avg(t2.rating) avg_rating, \n",
    "    count(1) rating_count \n",
    "from movies t1 join ratings t2 on t1.movieId = t2.movieId \n",
    "group by t1.movieId, t1.title\n",
    "having rating_count >= 100\n",
    "order by avg_rating desc \n",
    "limit 10\n",
    "\n",
    "\"\"\").show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Find average rating of each genre (advanced)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+------------------+-----+------------------+\n",
      "|             genre|count|        avg_rating|\n",
      "+------------------+-----+------------------+\n",
      "|         Film-Noir| 1210|3.9136363636363636|\n",
      "|               War| 5828|3.7832017844886754|\n",
      "|           Mystery| 8320| 3.652043269230769|\n",
      "|             Drama|46960|3.6502661839863713|\n",
      "|       Documentary| 1206|3.6430348258706466|\n",
      "|             Crime|18291|3.6423924334372098|\n",
      "|              IMAX| 3032| 3.641820580474934|\n",
      "|         Animation| 5966|3.6353503184713376|\n",
      "|           Musical| 4287|  3.57196174480989|\n",
      "|           Western| 2314| 3.565687121866897|\n",
      "|           Romance|19094| 3.544254739708809|\n",
      "|         Adventure|23076| 3.518027387762177|\n",
      "|           Fantasy|10889|3.5004591789879695|\n",
      "|          Thriller|29288|3.4955613220431574|\n",
      "|            Sci-Fi|16795|3.4544805001488537|\n",
      "|            Action|31205|3.4514500881269026|\n",
      "|          Children| 8098|3.4394294887626575|\n",
      "|            Comedy|38055|3.4209959269478385|\n",
      "|            Horror| 7983| 3.281097331830139|\n",
      "|(no genres listed)|    7|3.0714285714285716|\n",
      "+------------------+-----+------------------+\n",
      "\n"
     ]
    }
   ],
   "source": [
    "genre_avg_rating = (ratings.alias(\"t1\")\n",
    ".join(movies.alias(\"t2\"), col(\"t1.movieId\") == col(\"t2.movieId\"))\n",
    ".select(col(\"rating\"), explode(split(\"genres\", r\"\\|\")).alias(\"genre\"))\n",
    ".groupBy(col(\"genre\"))\n",
    ".agg(count(col(\"genre\")).alias(\"count\"), avg(\"rating\").alias(\"avg_rating\"))\n",
    ".orderBy(desc(\"avg_rating\")))\n",
    "\n",
    "genre_avg_rating.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Using matplotlib show barplot of average rating for each genre (Optional)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Loading matplotlib library"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Convert spark dataframe to Pandas Dataframe. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>genre</th>\n",
       "      <th>count</th>\n",
       "      <th>avg_rating</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Film-Noir</td>\n",
       "      <td>1210</td>\n",
       "      <td>3.913636</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>War</td>\n",
       "      <td>5828</td>\n",
       "      <td>3.783202</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Mystery</td>\n",
       "      <td>8320</td>\n",
       "      <td>3.652043</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Drama</td>\n",
       "      <td>46960</td>\n",
       "      <td>3.650266</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Documentary</td>\n",
       "      <td>1206</td>\n",
       "      <td>3.643035</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "         genre  count  avg_rating\n",
       "0    Film-Noir   1210    3.913636\n",
       "1          War   5828    3.783202\n",
       "2      Mystery   8320    3.652043\n",
       "3        Drama  46960    3.650266\n",
       "4  Documentary   1206    3.643035"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df = genre_avg_rating.toPandas()\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Plot average rating for each genre"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.axes._subplots.AxesSubplot at 0x7f3c6c9b6da0>"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXcAAAFkCAYAAADbgnvLAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvNQv5yAAAIABJREFUeJzt3Xe4XFW9xvHvSwgECD1RaSGCoCBSQkAFRIoFRUAQBRvlogEFjF7QKzaKHcVCEYyARkQERTQiKBB6N6EkQIKGokRQQgchQPB3/1hrkslkzpm9Z+bkTDbv53nmOTN79lqzzpw5v1l7VUUEZmZWLUsNdgHMzKz7HNzNzCrIwd3MrIIc3M3MKsjB3cysghzczcwqyMHd+iTpGEm/WEyvta2kv0l6RtJ7F8drDhRJX5B0+gDlHZJeMxB5W7U4uC9BJN0v6bkcAB+X9EdJ6wx2uWB+2d7WQRbHASdHxPCI+F23yjXQJO0gaXb9sYj4RkR8bLDKZAYO7kui3SJiOLAG8G/gpHYykbR0V0vVuXWBOwe7EPWU+H+kA34PB4/f9CVURMwFfgNsXDsmaVdJt0p6StIDko6pe250vqQ/SNI/gMvrjo2T9KCkhyQd0ddrStpd0p2SnpB0paSN8vGzgFHAH/JVxef6SP9xSbMkPSZpkqQ18/F7gPXq0i/bJO3nJd0j6WlJd0naMx9fNpdnk7pzR+YrnFfkx5/Lv9uDkj7WX9NG/r2+Luk64FlgPUkHSpqRX/teSQfnc1cALgbWzOV+RtKa9c1Zde/x/pL+IekRSV+se73lJE3MV2IzcllnNytbnXfncjwi6TuSlsrvw2OS3lCX9yvy+zCyye85RNIJOY/7JB2Wy7l0fn5lSWfk9+2fkr4maUh+7gBJ10r6bi73fZLe1eI97DM/GyAR4dsScgPuB96W7y8PTAR+Xvf8DsAbSF/am5Jq9u/Nz40GAvg5sAKwXN2xc/KxNwBz6l7jGOAX+f6GwH+AtwNDgc8Bs4BlGsvWR9l3Ah4BxgDLkq44rm72u/WR/v3Amvl32yeXZY383JnA1+vOPRT4U76/C/Av4PX5PTsr/86v6eN1rgT+kc9fOv+uuwLrAwLeSgpYY+re89kNedS/b7X3+Cf5Pd8MeB7YKD//LeAqYFVgbWBaY34NeQdwBbAa6Qv1r8DH8nM/Ar5dd+544A995HMIcFd+zVWBy3LeS+fnfwf8OH8uXgHcDBycnzsAeBH4ODAE+ATwIKB+3sM+8/NtgOLFYBfAtxJ/rBQAnwGeAOblf6g39HP+D4Dv5/u1ILNe3fO1Y6+rO3Y8cEa+Xx+kvgycV3feUsA/gR3qytZfcD4DOL7u8fAcIEYXSd8kv9uAPfL9twH31j13HbBfvn8m8M26515D6+B+XIvX/h0wPt/foTEY0zy4r133/M3Avvn+vcA76577WGN+DXkHsEvd408Ck/P9NwIPAEvlx1OAD/SRz+X1wTW/h5GD8StJX0DL1T3/QeCKfP8AYFbdc8vntK9q9h62ys+3gbn1WrurtfbeiLgsX9LuAVwlaeOI+JekN5JqgpsAy5BqyL9uSP9Akzzrj/2dVINvtGZ+DoCI+K+kB4C1CpZ7TeCWuvTPSHo0p7+/VWJJ+wH/SwqWkL4cRuT7lwPL5d//X8DmwAV1rzulLqtmv3+jhc7JTQ5Hk65eliIFs+kF8qn3r7r7z+by18pX/3ply/f3nAcRcZOk/wBvlfQQ6YtsUh959Pe665Jq2w9Jqh1bquGc+b9PRDybzxte93zZ/KzL3Oa+hIqIlyLit8BLwHb58C9J/8zrRMTKwGmkpoSFkjbJrn7EzSjSFUGjB0n/pEDqKMvp/tlPvv2lXwFYvS59nyStS2rWOAxYPSJWAe4g/24R8V/gPFJt8EPAhRHxdE7+EKnpoabI6KL5v0tu/z8f+C7wyvzaF7Hgfe10WdV2ytff32si8BHgo8BvIvXNlH3dB0g17RERsUq+rRQRry9Qtpr696Ub+VlJDu5LKCV7kNpLZ+TDKwKPRcRcSVuTAl0RX5a0vKTXAwcC5zY55zxgV0k7SxoKHEH6h70+P/9vUqdoX34JHChp8xwwvwHcFBH3FyjfCqRgMQdA0oGkq5PG/PcBPpzv15f7QEkbSVoe+EqB16tXuwKaA8zLtfh31D3/b2B1SSuXzLe+fEdJWlXSWqQvsFY+m89fh9SuXv/3OgvYkxTgf97idcdLWkvSKsD/1Z6IiIeAS4ATJK2UO2zXl/TWcr/awORnxTi4L3n+IOkZ4Cng68D+EVEbQvhJ4DhJT5OC2HkF87yK1Dk6GfhuRFzSeEJE3E0KGCeROkZ3Iw3LfCGf8k3gS0ojV45skn4yqd3+fFKtcX1g3yKFi4i7gBOAG0jB9A2kdvX6c24idbKuSRrBUjt+MXAiqRNyVs4D0hdTkdd+GvgU6b18nPSFOanu+ZmkDul78+++ZpF86xwHzAbuI3Vq/qZA2X4PTCX1O/yR1J9RK89sUvNXANf0k8dPSAF3GnAr6WpkHulKEGA/0hfbXaTf+zek4bft6nZ+1kKtd9tehiSNJgWVoRExb3BLs3goDd+8A1i2F39nSZ8gdba2XauVdCbwYER8qUSadwGnRcS6LU+2JYJr7lZ5kvaUtIykVYFvk4YH9kRgl7SG0tILS0l6Lam564JW6frJbzSwF3W1+T7OW07SuyUtnZuDju7kda33OLjby8HBpDbze0jNDp8Y3OIsZBnS+O+nSaN+fk8ar16apK+Srkq+ExH3tTodOJbURHIrqd+mbH+E9TA3y5iZVZBr7mZmFTRok5hGjBgRo0ePHqyXNzNbIk2dOvWRiFhkvaBGgxbcR48ezZQpU1qfaGZm80n6e+uz3CxjZlZJhYN7XiL0VkkXNnluWUnnKi3nelMejmVmZoOkTM19PAumuTc6CHg8Il4DfJ80ltjMzAZJoTZ3SWuT1rT+OmllvkZ7kJY5hTSt+GRJCo+zNKusF198kdmzZzN3bl9rk1knhg0bxtprr83QoUPbSl+0Q/UHpM0ZVuzj+bXIy3dGxDxJT5JW/Huk/iRJ44BxAKNGjWqnvGbWI2bPns2KK67I6NGjqVvK17ogInj00UeZPXs2r371q9vKo2WzjKT3AA9HxNT+TmtWvkUOREyIiLERMXbkyJYjecysh82dO5fVV1/dgX0ASGL11Vfv6KqoSJv7tsDuku4HfgXspLw/ZJ3Z5PWg8x6MKwOPtV0qM1siOLAPnE7f25bBPSKOioi1I2I0aYnWyyPiIw2nTQL2z/f3zue4vd3MbJC0PYlJ0nHAlIiYRFqB7ixJs0g19kLrdJtZdYz+/B+7mt/939q1q/m93JQK7hFxJWnzWyLiK3XH55J2p29bkQ+G/9hm1muuvPJKlllmGbbZZhsATjvtNJZffnn222+/QS2XN8g2M2th3rx5LL1083B55ZVXMnz48PnB/ZBDDlmcReuTg7uZLdHe+9738sADDzB37lzGjx/PSy+9xH333cfxxx8PwM9+9jOmTp3KSSedxFe/+lXOPvts1llnHUaMGMGWW27JkUcusiskADvssAPbbLMN1113HbvvvjsbbrghX/va13jhhRdYffXVOfvss3nuuec47bTTGDJkCL/4xS846aSTmDx5MsOHD+fII49khx124I1vfCNXXHEFTzzxBGeccQZvectbePbZZznggAOYOXMmG220Effffz+nnHIKY8eO7dr74uBuZku0M888k9VWW43nnnuOrbbaismTJ7PtttvOD+7nnnsuX/ziF5kyZQrnn38+t956K/PmzWPMmDFsueWW/eb9xBNPcNVVVwHw+OOPc+ONNyKJ008/neOPP54TTjiBQw45ZH4wB5g8efJCecybN4+bb76Ziy66iGOPPZbLLruMH/3oR6y66qpMmzaNO+64g80337zr74uDu5kt0U488UQuuCDtEPjAAw9w3333sd5663HjjTeywQYbcPfdd7Ptttvywx/+kD322IPlllsOgN12261l3vvss8/8+7Nnz2afffbhoYce4oUXXig8uWivvfYCYMstt+T+++8H4Nprr2X8+PEAbLLJJmy66aaFf9+ivCqkmS2xrrzySi677DJuuOEGbr/9drbYYgvmzp3LPvvsw3nnncf555/PnnvuiSTaGZ29wgorzL9/+OGHc9hhhzF9+nR+/OMfF55gtOyyywIwZMgQ5s1LW/cujpHi1aq5H7Nyi+efXDzlMHsZGozRbE8++SSrrroqyy+/PDNnzuTGG28EUm3561//Ouuuuy7f/nZax3C77bbj4IMP5qijjmLevHn88Y9/5OMf/3ip11prrbUAmDhx4vzjK664Ik899VSpcm+33Xacd9557Ljjjtx1111Mnz69VPoiXHM3syXWLrvswrx589h000358pe/zJve9CYAVl11VTbeeGP+/ve/s/XWWwOw1VZbsfvuu7PZZpux1157MXbsWFZeuUWFsM4xxxzD+9//ft7ylrcwYsSI+cd32203LrjgAjbffHOuueaaQnl98pOfZM6cOWy66aZ8+9vfZtNNNy1VliIGbYPssWPHRv1OTF0Z5+6au9liM2PGDDbaaKPBLkYpzzzzDMOHD+fZZ59l++23Z8KECYwZM2axl+Oll17ixRdfZNiwYdxzzz3svPPO/PWvf2WZZZZZ6Lxm77GkqRHRclhNtZplzMz6MW7cOO666y7mzp3L/vvvPyiBHeDZZ59lxx135MUXXyQiOPXUUxcJ7J1ycDezl41f/vKXixw79NBDue666xY6Nn78eA488MABK8eKK6444HtIO7ibWdsiYolfGfKUU04Z7CI01WmTuTtUzawtw4YN49FHH10sw/pebmqbdQwbNqztPFxzb/CGiW/o9/np+3d/yJLZkmjttddm9uzZzJkzZ7CLUkm1bfba5eA+AGa8rv8RBBvN7Guf8eSUQy5v+RqHnrZTv8+fsM97+n3+iHMvbPkasz/f/7Cutb/1lpZ5HHPMMR09b71r6NChbW8BZwPPzTJmZhXk4G5mVkEO7mZmFdSyzV3SMOBqYNl8/m8i4uiGcw4AvgP8Mx86OSJO725R7eVo8uXrtzxn553u6ff5V11xW7/P/2vH7i+3ajbYinSoPg/sFBHPSBoKXCvp4oi4seG8cyPisO4X0WzwtVoew1tAWq9pGdwjDWJ9Jj8cmm8e2Gpm1sMKDYWUNASYCrwGOCUibmpy2vskbQ/8FfhMRDzQJJ9xwDiAUaNGtV1osyVRx7X/VgvjgRfHs/kKBfeIeAnYXNIqwAWSNomIO+pO+QNwTkQ8L+kQYCKwyEDsiJgATIC0KmTHpTezUjxJ7+Wj1GiZiHgCuBLYpeH4oxHxfH74E6D/jQnNzGxAFRktMxJ4MSKekLQc8Dbg2w3nrBERD+WHuwP9T8E0syVWL8zAttaKNMusAUzM7e5LAedFxIWSjgOmRMQk4FOSdgfmAY8BBwxUgc3MWi2vAcWW2KiyIqNlpgFbNDn+lbr7RwFHdbdoZmYDpxtrJ/Uyz1A1M6sgB3czswpycDczqyAHdzOzCnJwNzOrIAd3M7MK8jZ7ZmZt6uVtJF1zNzOrIAd3M7MKcnA3M6sgB3czswpycDczqyAHdzOzCnJwNzOrIAd3M7MKcnA3M6sgB3czswry8gNmZoNk8uXrtzxn553uaSvvljV3ScMk3Szpdkl3Sjq2yTnLSjpX0ixJN0ka3VZpzMysK4o0yzwP7BQRmwGbA7tIelPDOQcBj0fEa4DvA9/ubjHNzKyMlsE9kmfyw6H5Fg2n7QFMzPd/A+wsSV0rpZmZlVKoQ1XSEEm3AQ8Dl0bETQ2nrAU8ABAR84AngdWb5DNO0hRJU+bMmdNZyc3MrE+FgntEvBQRmwNrA1tL2qThlGa19MbaPRExISLGRsTYkSNHli+tmZkVUmooZEQ8AVwJ7NLw1GxgHQBJSwMrA491oXxmZtaGIqNlRkpaJd9fDngbMLPhtEnA/vn+3sDlEbFIzd3MzBaPIuPc1wAmShpC+jI4LyIulHQcMCUiJgFnAGdJmkWqse87YCU2M7OWWgb3iJgGbNHk+Ffq7s8F3t/dopmZWbu8/ICZWQU5uJuZVZCDu5lZBTm4m5lVkIO7mVkFObibmVWQg7uZWQU5uJuZVZCDu5lZBTm4m5lVkIO7mVkFObibmVWQg7uZWQU5uJuZVZCDu5lZBTm4m5lVkIO7mVkFFdlDdR1JV0iaIelOSeObnLODpCcl3ZZvX2mWl5mZLR5F9lCdBxwREbdIWhGYKunSiLir4bxrIuI93S+imZmV1bLmHhEPRcQt+f7TwAxgrYEumJmZta9Um7uk0aTNsm9q8vSbJd0u6WJJr+8j/ThJUyRNmTNnTunCmplZMYWDu6ThwPnApyPiqYanbwHWjYjNgJOA3zXLIyImRMTYiBg7cuTIdstsZmYtFArukoaSAvvZEfHbxucj4qmIeCbfvwgYKmlEV0tqZmaFFRktI+AMYEZEfK+Pc16Vz0PS1jnfR7tZUDMzK67IaJltgY8C0yXdlo99ARgFEBGnAXsDn5A0D3gO2DciYgDKa2ZmBbQM7hFxLaAW55wMnNytQpmZWWc8Q9XMrIIc3M3MKsjB3cysghzczcwqyMHdzKyCHNzNzCrIwd3MrIIc3M3MKsjB3cysghzczcwqyMHdzKyCHNzNzCrIwd3MrIIc3M3MKsjB3cysghzczcwqyMHdzKyCHNzNzCqoyAbZ60i6QtIMSXdKGt/kHEk6UdIsSdMkjRmY4pqZWRFFNsieBxwREbdIWhGYKunSiLir7px3ARvk2xuBU/NPMzMbBC1r7hHxUETcku8/DcwA1mo4bQ/g55HcCKwiaY2ul9bMzAop1eYuaTSwBXBTw1NrAQ/UPZ7Nol8ASBonaYqkKXPmzClXUjMzK6xwcJc0HDgf+HREPNX4dJMksciBiAkRMTYixo4cObJcSc3MrLBCwV3SUFJgPzsiftvklNnAOnWP1wYe7Lx4ZmbWjiKjZQScAcyIiO/1cdokYL88auZNwJMR8VAXy2lmZiUUGS2zLfBRYLqk2/KxLwCjACLiNOAi4N3ALOBZ4MDuF9XMzIpqGdwj4lqat6nXnxPAod0qlJmZdcYzVM3MKsjB3cysghzczcwqyMHdzKyCHNzNzCrIwd3MrIIc3M3MKsjB3cysghzczcwqyMHdzKyCHNzNzCrIwd3MrIIc3M3MKsjB3cysghzczcwqyMHdzKyCHNzNzCqoyB6qZ0p6WNIdfTy/g6QnJd2Wb1/pfjHNzKyMInuo/gw4Gfh5P+dcExHv6UqJzMysYy1r7hFxNfDYYiiLmZl1Sbfa3N8s6XZJF0t6fV8nSRonaYqkKXPmzOnSS5uZWaNuBPdbgHUjYjPgJOB3fZ0YERMiYmxEjB05cmQXXtrMzJrpOLhHxFMR8Uy+fxEwVNKIjktmZmZt6zi4S3qVJOX7W+c8H+00XzMza1/L0TKSzgF2AEZImg0cDQwFiIjTgL2BT0iaBzwH7BsRMWAlNjOzlloG94j4YIvnTyYNlTQzsx7hGapmZhXk4G5mVkEO7mZmFeTgbmZWQQ7uZmYV5OBuZlZBDu5mZhXk4G5mVkEO7mZmFeTgbmZWQQ7uZmYV5OBuZlZBDu5mZhXk4G5mVkEO7mZmFeTgbmZWQQ7uZmYV5OBuZlZBLYO7pDMlPSzpjj6el6QTJc2SNE3SmO4X08zMyihSc/8ZsEs/z78L2CDfxgGndl4sMzPrRMvgHhFXA4/1c8oewM8juRFYRdIa3SqgmZmV140297WAB+oez87HFiFpnKQpkqbMmTOnCy9tZmbNdCO4q8mxaHZiREyIiLERMXbkyJFdeGkzM2umG8F9NrBO3eO1gQe7kK+ZmbWpG8F9ErBfHjXzJuDJiHioC/mamVmblm51gqRzgB2AEZJmA0cDQwEi4jTgIuDdwCzgWeDAgSqsmZkV0zK4R8QHWzwfwKFdK5GZmXXMM1TNzCrIwd3MrIIc3M3MKsjB3cysghzczcwqyMHdzKyCHNzNzCrIwd3MrIIc3M3MKsjB3cysghzczcwqyMHdzKyCHNzNzCrIwd3MrIIc3M3MKsjB3cysghzczcwqqFBwl7SLpLslzZL0+SbPHyBpjqTb8u1j3S+qmZkVVWQP1SHAKcDbgdnAXyRNioi7Gk49NyIOG4AymplZSUVq7lsDsyLi3oh4AfgVsMfAFsvMzDpRJLivBTxQ93h2PtbofZKmSfqNpHWaZSRpnKQpkqbMmTOnjeKamVkRRYK7mhyLhsd/AEZHxKbAZcDEZhlFxISIGBsRY0eOHFmupGZmVliR4D4bqK+Jrw08WH9CRDwaEc/nhz8BtuxO8czMrB1FgvtfgA0kvVrSMsC+wKT6EyStUfdwd2BG94poZmZltRwtExHzJB0G/BkYApwZEXdKOg6YEhGTgE9J2h2YBzwGHDCAZTYzsxZaBneAiLgIuKjh2Ffq7h8FHNXdopmZWbs8Q9XMrIIc3M3MKsjB3cysghzczcwqyMHdzKyCHNzNzCrIwd3MrIIc3M3MKsjB3cysghzczcwqyMHdzKyCHNzNzCrIwd3MrIIc3M3MKsjB3cysghzczcwqyMHdzKyCHNzNzCqoUHCXtIukuyXNkvT5Js8vK+nc/PxNkkZ3u6BmZlZcy+AuaQhwCvAuYGPgg5I2bjjtIODxiHgN8H3g290uqJmZFVek5r41MCsi7o2IF4BfAXs0nLMHMDHf/w2wsyR1r5hmZlaGIqL/E6S9gV0i4mP58UeBN0bEYXXn3JHPmZ0f35PPeaQhr3HAuPzwtcDdLco3AnikxTkDmb5KefRCGbqRRy+UoVfy6IUy9EoevVCGxZXHuhExslUmSxd4oWY18MZvhCLnEBETgAkFXjNlKk2JiLFFz+92+irl0Qtl6EYevVCGXsmjF8rQK3n0Qhl6KQ8o1iwzG1in7vHawIN9nSNpaWBl4LFOC2dmZu0pEtz/Amwg6dWSlgH2BSY1nDMJ2D/f3xu4PFq195iZ2YBp2SwTEfMkHQb8GRgCnBkRd0o6DpgSEZOAM4CzJM0i1dj37VL5CjfhDFD6KuXRC2XoRh69UIZeyaMXytArefRCGXopj9YdqmZmtuTxDFUzswpycDczqyAHd7OXCUlLSfrAYJfDFo+eCu75w7dNB+klaZ3WZ7bM5z2SBv296VY5JK3QjfJYMpjvZ14OpC0R8V/gsJYn2mKV494WknaVtJOkV3Yl317rUJV0Q0S8uYP0UyNiyw7L8AvgzcD5wE8jYkYbeWwAfJO0Hs+w2vGIWG9xlSN/UZ4ODI+IUZI2Aw6OiE8WSHsE8IOIeKnh+OrA8RFxUIlyDAFeSd3orIj4R4F0f6DJZLi6PHYvUYZtgWOAdXM5lLIo9fdo+/3M6QV8GFgvIo6TNAp4VUTcXKIM95GW+PhpRNxVNF1d+i8DzwHnAv+pHY+IxTovRdJI4OPAaBb+XPxPyXza+mzVpT8MODsiHi/zug2v/+eIeFsbadcH/g94G/A3YA4pVmwIPAv8GJiYv5TLi4ieugHHAu8jf/G0kf4UYKsulGMl4GDgRuAG0rIJK5ZIfy2wMzCNFFCOAY5dnOUAbiJNLru17tgdBdNOAG4Dtq079kngPuDTJcp/OGkq9Z3A9HybVjDtW/u7lXwfZ5IWv3sFsHrtVjKPtt/PfO6p+fM5Iz9eFfhLyTKsSAqK1+fPxDhgpRLp72tyu7fM5zr/fBp4qu72NPBUiXyuJy0w+IH8//4+4H0l34u2P1t1eXwNmAWcB+zSTtwhzfNZuY105wDbN3vN/Dn9NLB/2Xzn59FuwoG65Q/Jf4EX2vzQ3AXMA+7JgbX0H7wurxH5Db4fuJj07Xp4wbRT88/pdceuWZzlAG7KP+uD0e0lXncb4BbgLNJktl8Ca5Qs+6yyQXSAPlc3dSuPDt7PWzpJ3yS/7YF/kmrgE4HXDPb7XKLst3Uhj658tkhXce8kLYo4C/gGsH6J9OcB/yDN9zmxdhvs97jI2jKLVUSs2GEW7+q0DJJ2Bw4E1icFtq0j4mFJywMzgJMKZDM3t5f/LV/6/ZP0bbw4y/FAbkqIPLv4UzldUXeQgvoupH+AIyLioTK/A/AA8GTJNAvpRhMXcIWk7wC/BZ6vy+OWEnl0+n6+mC/jU0RJTROlLrlz+l1Jn4vRwAnA2cBbgItIl/T9pV8e+F9gVESMy+/tayPiwoKvv1dE/DbfXzXabM4ALpT07oi4qM300IXPFqS2OUn/Av5FqhiuCvxG0qUR8bkCWfwx30qRtFeLcv22bJ4L5Z+/eQadpNdFxExJY5o9X/KfEEmvYOFAUKYdbiJwRkRc3eS5nSNicoE8tiL9468CfJW03s7xEXHj4iqHpBHAD0ltegIuAcZHxKMFXvsjwHGkdr8TgM1ITQp/BY6MiIcL/g5nkFYA/SMLB9XvFUmf87gWOJq0V8BupMCmiDi6RB5XNDkcEbFTiTzafj9z+g8D+wBjSDXtvYEvRcSvS5ThXuAK0ufi+obnToyIT7VIfy4wFdgvIjaRtBxwQ0RsXvD1b4mIMY33y5L0NLAC6Qr9xXw4ImKlEnl047P1KdLSKY+Q+lN+FxEv1ipmEbF+wXyWYcEX690R8WJ/5+c0P813X0G6Sr48P94RuDIi+g3+LfPvoeA+IdckOvonzLXdE4A1gYdJ7d0zIuL1BdO33UHSTYNdDkm/Bz4VEX+vOybgEOCzRWvNkpoG4Ig4tkRZpkbElpKmR8Qb8rFrIuItBdMvBewdEecVfc2BIul1pL4YAZOjfCf58Ih4poPXnxIRYyXdGhFb5GO3R8RmBdPXp5t/fzB06bN1HOmL8u9NntuoyN9H0g6kL+v7SX/XdUht5YtUyvpIfyHw8dpVsaQ1gFM6De6D3vbW7RtwO6mz7Nb8eEdgQsk82uogachjLHABqc16Wu22OMsBvBr4HqkpYlLt1oX3eFQbaVYkjTJp5/WuIw3b/S1pKN+epNpRmTyu7sLvPRFYpe7xqqS1loqkXYoSna/95HM8qZN9KDCZVOP8SIn01wPLsaD9f33g5hLpZwJbAFuSrky3IF2JjAHGlPxddge+m2/v6eA9WaGNNKv1dysV675OAAAeUUlEQVSZ11RS01bt8YbkPreC6e9oeNyVz0rPtblLGgp8gtRZBHAl8OMocJmTvRgRj+axo0tFxBWSym77NxeYLulSFh4u1u8lb4Ozgc+SOnTbG8rUeTl+R+rk+UMHZQBA0sqkEQ0fAjYC1iqYbhNSf8Fq+fEjpCaBO0u8/KeB5Ult3F8lfWHv32+KRV0q6Ug6GwK4aUQ8UZf2cUmFaq4R8V9Jt0saFSWaCJt4R0R8TtKepKW2309qpvlFwfRHA38C1pF0NrAtcECJ13+IVGGA1EZd3wQSQNEr7G8BW5H+TwDGS9ouIhbZo7mfPN5M+nwPB8oOTZ2ayytgFPB4vr8KqXP01UXLAQyNiPkbD0XEX3McK+pKSX8mjZ4J0sKLzVowSumZZpkaSaeTaiW1bfs+CrwUeSeoAukvA94LfItUg3+YNDSy8OQoSU0DR0RMbHa8jzyujYjtip4/EOWQdFNEvLGD11+OVLv6EKlmtiLpvb06Co69lXQ98MWIuCI/3gH4Rpm/Rzfk8eGNIsqNc78d2CFyJ6Kk1YCrIjcVFUh/OSmg3czCXzBlxuvfGRGvl/QT4PyI+FOZZpWcx+rAm0jB7MZo2DFtcZA0Ddi89jnKzZC3RsSmJfK4idRvMSkWNBXdERGblMjjtJz+ovz4XcDbIuKIEnmcSQrKZ+VDHwaWjogDS+SxJwsqtFdHxAVF0/aZZw8G90U+qEU+vJI+Tbp8n0GaALAU6U1emTRJoVCnV11+y5GaH1ptBdhX+p2BD5Iunes7ezrqAS9Zhg8BG5A6/kqNEMm1uu1z2l+ROntmRUSZGk3bf8+G8y8F3l+rNUtaFfhVRLyzTFk6JWk/4CjSJCJIteavR8RZfadaKP1bmx2PiKtKlOFbpC/Y50j7G68CXNjqS7yvgQp1ZSg1YKEh7wkRMa71mQulmUb6onwsP16N1IlYKrhHxBvb7T/I5y8y6VEld0KStCxwKLAd6QvzauBHEfF8vwkXzmNdYIOIuExpRNOQiHi6aPpmeq5ZBnhJ0voRcQ+ApPWAl1qkgbRD1A+B15Hat68nBfs/lLz0RtJupHbAZYBXS9ocOK5MDYs0ouN1pKuQWi03SO3GRcvR6RDAN5CufHZqKEORS+dNSJeqM4CZEfGSpHZqAvcqzYqsBcCPkCbOlDGiSXNI2WGlHQ0BzK/7c0lTSc1CAvaKErNEywTxfvL4fG5mfCr/Tf7DohvWN3NCf9lSsDmlD+1sCfdN4NY8gEKkisRRJfPodGgqwCOSvkRq1grS57NwRTBfcZwRER9h4SaqwiR9nDQZbTVSH8hawGmkjvf2ddpo3+1b/oX+QWprv4rUA71jifTLkIYVHUmatv8gcFfJMkwl1fjrJ5tML5lHqfP7yKOjWa6kzq9lOnj915GGQ94NXEOaHv2qknmsSprUcQtwK+kLeNU2/h6j6h6vS+4QLJHHucDnyB1VpE7F0hNpSBvWrElqpx1Fic5lFp7VOZdUaSk8Qa8un01IMzv3q906/ax1+Dn9U5vp1iA1++1R9nOV048gtdn/m9T8+gvKzzpeLX8mb82f0R9QvkP1zx3+n92W41bb8abZredq7hExuVarIn2jz4wSlzekf9qVSMF5ZVJwn16yGPMi4sk08m9B0UrmcaOkjaON9T/qLJffD0UaqnWMpGtInWJF3E66bC80Jr1RRMwEvgJ8RdJYUtv7zZJmR8E280jt02U6opv5InCtpFrNd3tSTaeM9SNiH0kfzOV6Tg1/4FYkHU567/9NCswifS4KNSVEwwQ9Se8lNa2UKcPRwA6kq7mLSJP2rgV+3iLdgE2YiYhdip6rReezzM4/15S0ZhRsHso15o9GxIdLFnchka7qx3c4xPR+4DpJk1i4L6VoTf75iHih9nFU2oe64/byngnukrbv46k3SiJajBmVNAF4Pal2dBOpWeZ70d4Mujtye/WQ/EXzqZxfGdsB++eOvOdh/kJVhdsU6XyW6yuBmZL+wsJt7mWal2pppgBTlBYU6+tvNZ+kH0TEp9XH4l9lyhCp03AMCzoBPxPlOwFfyP0otdmh61P3nhQ0ntSUU6r/pi8R8TtJhUeHZHuTJpTdGhEHKq0geHqBdLv1VxQKNhd24e/6v6Qv5mbNRIWbhyI1Se1BmtjWNtUtBkf5ETc1D+bbUqRBB2VdJekLwHKS3k5aw+kPbeSzkJ4J7qRhg42C9EFem3Q53J9RwLKkdVf+SaoRPNFvir4dTqotPk9aT+XPpCF4ZRSuzfSj2RDA/UqkLzyDs5GkE1uc0qr9uNbG/t0OytBYy3sw/xylNKSwTCfgMSw6BLDwaIaso+nuDbXnpUht1WVraM9FGlY5T9JKpKuyln0wUWLkRgsd/V1jQcfruyJibv1zkoY1SdKf6ySdzKLDW8t8Lr5PWldmUk57ez8VzUXkK4jhEdEsfhX1eeAgUgvDwcBFEfGTDvIDeii4R8RCNQtJ25EC7EMUWIM6InbJl9mvJ7W5HwFsIukx0vTqMoFu14j4Yn79WnneDxSeJp6bUVDDMggljY6IvwDPkANRLsdNBcvQSQfeIaS1Zc4jBdVSTRgRMTXf3Twiflj/nKTxtP5ygC7V8nJ5LsmdobXa//g2av/3ksYktzvdvf4zPo90OV+kM7TeFEmrAD8h9UU8Qxpa2S9JH4mIX0j632bPF/0d6v6uU8hfNDn/IaTKVVHXk4bXtjrWn1rT4HH1RaRk53BEPNDQQldkAEct7UutRiIVcHj+H5kf0CWNb/y/KasXh0LuDHyZ9Ef6RkRc2kYea5NqZtsA7yF1sqxSIv0ia2Y0O9Yij46WQeikHMpj7JXW76j/A9eahlqu36E0Fvr9pLVQ5pFqR+eXbebq43dY7NPWJU2OiJ1bHWuRR0fT3SVtGxHXtTpWojyjScv9Titw7sER8eNOf4e6/G4kjQd/Jj8eDlzSqi9G0qtIo0F+QerDqUXVlYDTIuJ1ZcrRKUm/IY1yOZn0xf8pYGxE7FsijxNIQ45/zcJXEEWbugbkf6Rnau6SdiXVlJ8kTXop9YFXWgBoG1JQf5E0DPIG4EwKdqgqTWB4N7BWQ7PESqQAV8ZXSR+WyyJiC0k7ksa9D3g5Ik+eauzAKyO3K58GnCZpLVLZ75T0f1FgXHfuuPwQaSjppLqnVqTEULOc1/tJIzKezsPWxgBfjYhbC6QdRmraGqE0Pr4+mKxZphxlA2ATJ7FozbTZsT7VfyFFxP2Nx/qSA/sQ0uicjtqps2H1HZAR8YzScNNW3kmaEbs2qfJT+3s8BXyhyAv3dfVRV5YyQxIPIY2WWYvUlHsJacx6GauRPtP1Vwwt+zH6+R9ZiZL/I830THAndSDMJv1S/9c4kKFAR81o0uSSz0T5ZWlrHiRdbu5OuuSteRr4TMm8OlkGoeNy5I7YaVFitl4f+YwhBfa3k9aSn9p/ivmuJzWpjWDhZpWnSUM7y/hyRPw6N9W9k9TeexpQZPbtwaS+izVJZa8PJqeUKYTSEr2fIzX91c876LcZQGma/DbAyIbAtBKt+5JqeXT8JZWbEHanw07I7D+SxtTat5VGUz1XoAwTgYmS3hcR57f52rVKy2tJM35rgXE30gSiwnLTXKcjbtrtz+jm/8gieim479hJ4ojo99u8YB63A7dL+mXktWzyP9I6bYy6eSJfql4NnC3pYQrW/puVo6zocC0TSceSmrRmkGaoHhURha9ecp/D30nbBHaq1ga6K3BqRPxe0jEFy/FD4IeSDo+IIuvw9+dsUvPUe0g1vv1JY/9bWYY0GmNpFh5N8RRp9EsR3fqSur4LnZDksvxa0oOkWuqapCa8orbMVxz1s46PiIgvtUpYu4KSdAlpsbKn8+NjKNEvltO8mjSAYjQLb9VXZkmItUlXYNuS3otrSX06s/tLV/sfkfQ2FnSUb0iaX1J2+Pai5eq1Nvd69TWDxfy6V5JqzUuTJhjMIa0hUvgLRGkT5efoYBkEdbjvpzpYy0TSf0kdiLXaWO2DUmpIp6Q3kT74G5GC3BDgP0Xa/evyuJA0AuptpNUInyOtZFhmmnmzpp2vlfl8acHSw9Nqv7+kqyKi6bICTdKvG02Wli2j0y8pdb6k9lbAAxHxL6XFsQ4G9iLtgPaVKDgbvFmbchv9WjOBzSLPg1FaBuD2Mu32SusFnUHDAn9lBiMoLY/xSxaehf3hiHh7wfRTSZutrEraOnEK8Gx0OIa/l2ruzZxOud7zblk5Ip6S9DHSRsRHK62FUUhu2/x9pLXY/8uCRdDKOoPUDDOVEj34kl5DGuPe2Eb8VlKQLKLUGjL9OJm0yt2vSUP/9gNeUzKPD5CGln43Ip5QWu+67NCzZk07p1KsaaemdhX1UO4jepDUdlzUskrzMUazcC2xzKifk5TGZjfm0e8kprrzOrpCJm3eUttj4M2kdvLDgc1J++4WvRIZImnZusC8HOVG20AKpjdLuoBU+diTFpO5mpgbEa2G/bYyMiJ+Wvf4Z0prXRWliHhW0kHASRFxvKSW/Umt9HpwLzX8rouWzgHkA9QNhywqt20+K2nliOhkG7AnI+LiNtL9APhC4ygKpXVIjiZ9afSr0xpmQ16zJA2JiJeAnyqtFFnGCFJtBkmj8rGZJfNou2mnzteUlj4+gnQ1shLl+mJ+TeorOJ0SX9b1JJ1FWn/ktro8goJBLddu38eiXw7H9ZWmwZC62vk+pL0SzgfOl3RbwTwgjZaZrAW7ER1IyUpQRHxd0sWkWi/AgUU62Rv8MI8gKr24Xp1HlHYuOyc//iDlOkSV+2U+TBrvDl2Izb0e3DsdndCu40gTl66NiL8oLV72t5J5dGNN+Hb3/RzdbHhcRExRGj7XkhYdRjn/KQoOp8yeVVrU6TZJx5M6kFYomLbmjyxYe3sY6ariblLHZlH/lFSrdX47B7mlyhQiFiwy9iTt9RHNi4hT20hXbyywcbTfnvp7UvmnUn6GLqQa99K5/2VnFl4GonA8ybXTaSzYsvBPpObHlpRWkKy5P9/mP1e0aSjrZHG9mv8hXaF+P6e9Ph8r6tOkRdMuiIg7c7yp3nruAJI2ZdGaxeJcKrfsB6RZHt1YE76t9lFJsyKiadNHf88NBKWlTB8mrY75GVLfw48iYlYHeY4hTRE/uESa5UlNO9Mj4m/5yuwNEXFJiTw66nzLVwoPk3boqv+yLvxZk/Rr0vaHbY0IU8n1zpuk/yJpmO4jpFnhYyIiclPgxIjYtkRem5OGAn6AtFLo+RFxcoF097Hgyx4W7Q8qs0b/TNImLC8UTbOk6LngrrTw/abAndR9k0ZEmW/CTsvwN9Jl70+Bi9utJeWhc0REkREVXSPpHODyaJjCnNv03hERZUY19KSynW85zXakNbN/mv82wyOi8PLDnXa+qTsbhlxBat++mTbWC8pt/idFRNujMXIn+RqkSUv/ycc2JL2f/V5V5vP2ZUHTxbmkDdcL1dq7TWnD8MOj4IbvDWlPop/lI1pdpauL6y81zb8Hg/tdEbHxIJdBpMvF/yGt2ncu8LOI+GvBtEeTlkwQ6dJ/Hukfqmi7Zi2vVwLfANaMiHdJ2hh4c0T022ae011A2lm+Ni59LGm0yp4R8a8y5eiEpPeQJnQ1jvgpM1qmfpTSUqRO9tWjxGYduV11LGnhrw0lrQn8umRNs6OdrbpBbW74IekO0hfS0qTZlPfS/oJ2bcujsK4BDqpdvUm6t+QXXOOaQwspOQLqSlJlsvTieg1X58fSsJZTq6t0SVtGxNR2/6Yty9eDwf0M4ITobKncrlGaWfoLUjvx7cDnI+KGfs7/DOmydVytVpjb0E4lDcUrPIEkdxb9lDRjdzOlpUBvjeLbuu1IWvsb4M6IuLzoa3eLpFmkoXLTO7gCqv+nqa3Jcn40LDzVIo/bSJs53xILdu2ZViaoqYOdrXL6jjcMaZekx0k1/qa62YHeohx7kmru25Da2X8FnB4ldvhS3vmp3WbLhry6Elg1CEtqtNKLwX170mzVfzEINYtchtVJY1U/Slq7+wzSLLjNSbW9Pj+IeQjT26NhUarcDHBJmQ+ApL9ExFZaeBux2yKiz3/SXpP/AXeOgnuuDmA5bo6IrWvNOUrzEG4oGdy/SfpM3MPCTYZFx4ifS7qS2i8iNlEa/ndDmb+n2pw30E4z1kDK7/97Sc0zO5FGylxQpg+ki2V5JWk+CKT5E+000bTTTDid/pt1Oop5vTha5kzSP9BC7ZqL2Q2kMbTvjYVnmU1R2lC3P0MbAzukdneV2xEd0hTv1VmwBvmb6GDJ2UHyOeAipY02Sq2kqIXX21hEyTbJ8/JomVWUtjX7H+pW4StoT2C9DjrfOt4whObzBjYokO4V6mdNliJ/j27KbfVnk2Zvr0ZapO7zpKuiwtTBmP+c/gPAd0g7vwk4SdJnI+I3/SbsjvcMZOa9GNz/ERH9/lMvBq/tqwkhIlqtD9PfP37ZoPC/pCuG9SVdB4yk+CSRXvF10rK0w0g1zTLeTFpD/RzSMsel5z1owcbpPyANX3yKtCbJV6L8iqMd7WxFdzYMaXfewBDSEgiDNXekT3m00I/zrTB1OOY/+yKwVa22nq+wL2PBJuj9vX79cOHlJT1Ve4oC/UoD3RTWi8F9pqRfkppm6mt6Az4Usr6m2KxCVbCmuFndH3mh7Cm5rntE3JLbBGtbDt4dba41M4hWi4h3tJn2VaQFy2qr5/0ROCci7iyRR18bpxddAK1epztbHcOiG4YcULIM7c4beKhsh/4SoNMx/wBLNTTDPErB+Q/Rwaqri0Mvtrn/tMnhiMUwFFLSHPqpKXbae91GeYaQZlSOZuHLzsV6Cd0JSd8iDcvsqC1VadLRB0mX0MdFyfVVckAcS+rIe3O+PVFmZFY3Ot9yM1ttw5AbmzXhtUi/LqkfaBlKzBvoxQ6/TqnDMf85j++QRsvUZpfuQ1pN9f+6UMRB1XPBfTDlYFqrKW5KezXFbpbnIvJMVxYeVz1YM3dLy5euK5Bqui9ScihkDuq7kv4mo0nNVGdGRNE1cmr5rEwK6Nvmn6uQRvCUWq61k863fGV4DjAptzmXed22VvesS9/xxLxeUTcufEXaHPOvvP5SRFyntP3hdqTP5uOkBf7uGYiyFyhXbRXajpf87Zng3umEgG7rtKbYpTKUGqpXNZImkoZyXgz8KiLuaCOPxo3TbyTVmEtvnN6k8+0tQOHOt1zz34f0ZXUzaf7EhUWGdNaPxpB0fkS8r2z5qyJ3iL+SNF6+3luBf0aLeSA5jwtpvv7SWODoaNj2cyCpC6vQNtNLbe5TBrsA0LSmeCIFd4YfABdLesdgDA/rVJcmmnyUtC7PhsCn6vpBytT+u7lxetudbzC/+eaqfIW4E/Bx0uiwIr9HfRNh4Qk/FbUHHS6MRxfWX+qijlah7UvPBPcosebKQGmoKR7bTk2xy24ELlDaVal0k8Yg63hz64gotbBXH3l0c+P0tjvfavJomd1INfgxFF8JMfq4/3LUjcDc3+CG5dopVAc6WoW2L73ULDOg6ywULMN/WbCCY1sbS3e5PPeSJnq0PbvTFlDnG6c363ybHhGfK5j+XNL68X8CzgOuLDq5S9JLpM+mSMHn2dpTLDlf+F2hLiyMpx5af0lpI5kvA9dFxCeUZrR/p9Omt14K7gO6zsKSSNKfgXcVDQC9qtOJJh2+dl8bp19HCsyl3tuGzrerI+KCEml3AS7N49OtTd0IzOqh9ZcGSi8F945GA1SRpJ+R2lcvpuTszl7R10STxdVBLul75LHtnQyZ6yPvIcC+EXF2i/P26u/5xTGHo0q6GZjVG+svbUhae+qVkZal2BTYPSK+1lG+PRTcPRqggRZeMGu+JWwo5Aw6n2gyqCStBBwKrEUainlpfvxZ4LaI2KNF+trcjVeQriJqAWRHUtNMv8HfmuuFwNwNSktzfBb4cSxYQ6qjdfehhzpU8WiARSxJQbwfd5Bmmna11ryYnUUa/3wD8DHSP+IywB4R0XJrudpY+jz8buPaFUTuRDtloApddRFxBV3YsagHLB8RNzfMip/Xaaa9FNw9GqCB0oqKzTqXy2wBNthGAHdJqp9oEq1quz1mvcjLLEs6nbwLUUQ8XTKf0Q1NQ/8mLS1hL2+P5HWGamsO7U0XKkO9FNxra7IIWK7sIjwVdWTd/WGkjY07/kZfzI6puy9SZ+QHB6cobZu/nk+kzc/vayOwA1yZO8nPIf0j7wtM7lIZbcl1KDABeJ2kf5K2HPxwp5n2TJu7FSPpqohoOqKoV2nRvTJ/OxgzfttVNwwRFh6K2M6uUnsC2+eHj5M60Q7tYnFtCZLnsOwdEecprXG/VJsVh0X0Us3dGmjhXd6XArYktV/3PDXfK1MRseOgFqwNETGki9ndR1rbZv6m0F3M25YwEfFfSYcB55Vdb6gVB/feNpUFu7zPIwWDgwa1RMXNJK39sVss2CvzM4NbpMFRpS86GxCXSjqS9LmYH+A7XejNzTI2INSFvTKrQl3YFNqqS9J9TQ5Hp5+PjtfusIEj6VBJq9Q9XlXSJwezTEVFxAV5puDrSKsofgZ4paRTJbW7eceS6n2kPYGvkPQTSTtD7+2IZIMjIl7d5NbxF79r7j1MTTbD1hK86YIW7JW5zxI2nLMr1EObQltvGYglOhzce1he9nOz2uzOPN19WkS8fnBLZp16uX/R2QIDtUSHg3sPy6sQjgZOI3WsHgI8EBFHDGa5zKx7BmqJDgf3HpbHwB4M1NpoLyF1SnpVQbOKUBf2gm2ar4N7b1Pa2Pm1pJr73RHxYoskZrYEycuMtLUXbH88zr2HSdqB1Ol2P6nmvo6k/SPi6sEsl5l11TEDkalr7j1M0lTgQxFxd368IXBORGw5uCUzs26StC6wQURcJml5YEinyxB4nHtvG1oL7AAR8Vdg6CCWx8y6TNLHSZus/zgfWgv4Xaf5ulmmt02RdAZpPXFIK8VN7ed8M1vyHApsDdwEEBF/k/SKTjN1cO9tnyD94T9F3rMT+NGglsjMuu35iHihtlmHpKXpwp4WbnPvcZJGAkTEnMEui5l1n6TjgSeA/YDDgU8Cd0XEFzvK18G99yh9hR8NHEaqsYs0c+2kiDhuMMtmZt2V57McBLyD9L/+Z9J8lo6Cs4N7D8pL474bGBcR9+Vj65F2SP9TRHx/MMtnZr3Pwb0HSboVeHtEPNJwfCRwyZK6cJiZLUrSdBZtY38SmAJ8LSIebSdfd6j2pqGNgR1Su7skD4U0q5aLSc2uv8yP980/nwJ+BuzWTqYO7r3phTafM7Mlz7YRsW3d4+mSrouIbSV9pN1MHdx702aSnmpyXMCwxV0YMxtQwyW9MSJuApC0NTA8Pzev3Uwd3HtQlzdkNrPe9jHgTEm1gP40cFDe3OWb7WbqDlUzsx4gaWVSTH6iK/k5uJuZVY8XDjMzqyAHdzOzCnKHqpnZIMpzVz4BbJ8PXQWc1umua25zNzMbRJJOJ+3TMDEf+ijwUkR8rKN8HdzNzAaPpNsjYrNWx8pym7uZ2eB6SdL6tQd5kcCXOs3Ube5mZoPrs8AVku4lzUJfFziw00zdLGNmNsgkLQu8lhTcZ0bE8x3n6eBuZja4JG0DjKauNSUift5Jnm6WMTMbRJLOAtYHbmNBW3sAHQV319zNzAaRpBnAxp1uq9fIo2XMzAbXHcCrup2pm2XMzAbXCOAuSTcD8ztSI2L3TjJ1cDczG1zHDESmbnM3MxsEktSqnb3IOX1xm7uZ2eC4QtLhkkbVH5S0jKSdJE0E9m83c9fczcwGgaRhwP8AHwZeDTxB2iN5CHAJcEpE3NZ2/g7uZmaDKy/7OwJ4ztvsmZlZn9zmbmZWQQ7uZmYV5OBuZlZBDu5mJUkaMthlMGvFwd0qT9KXJc2UdKmkcyQdKWl9SX+SNFXSNZJel8/9maQTJV0v6V5Je+fjO0i6QtIvgen52Eck3SzpNkk/dtC3XuLgbpUmaSzwPmALYC9gbH5qAnB4RGwJHAn8qC7ZGsB2wHuAb9Ud3xr4YkRsLGkjYB9g24jYnLRU64cH8ncxK8Nry1jVbQf8PiKeA5D0B9JEkW2AX0uqnbdsXZrfRcR/SYs5vbLu+M0RcV++vzOwJfCXnMdywMMD9luYleTgblWnJseWAp7INe5m6rc4q0//n4bjEyPiqA7LZzYg3CxjVXctsJukYZKGA7sCzwL3SXo/pMWZJG1WMt/JwN6SXpHzWE3Sut0suFknHNyt0iLiL8Ak4Hbgt8AU4ElS+/hBkm4H7gT2KJnvXcCXgEskTQMuJbXVm/UELz9glSdpeEQ8I2l54GpgXETcMtjlMhtIbnO3l4MJkjYmdaROdGC3lwPX3M3MKsht7mZmFeTgbmZWQQ7uZmYV5OBuZlZBDu5mZhX0/xNeQKt47zPYAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f3c6ca48d68>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "df.plot(\"genre\", \"avg_rating\", \"bar\", title = \"Barplot of avg rating by genre\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Exercise 2: Use stocks.csv for the following execise\n",
    "1. Load stocks.csv as stocks dataframe with scheme inferencing enabled. Cache the dataframe.\n",
    "2. What are the data type for each columns?\n",
    "3. Cast the date field of stocks dataframe as date type\n",
    "4. What is the largest value in date column. You should get 2016-08-15. \n",
    "5. Register the stocks dataframe as stocks temporary view.\n",
    "5. Create a new dataframe, stocks_last10 with last 10 records for each stock. Select date, symbol and adjclose columns. You can use SQL statement and SQL window operation. [Hint: if you are not familiar with window operation in SQL, please refer this - https://databricks.com/blog/2015/07/15/introducing-window-functions-in-spark-sql.html]\n",
    "6. Create a new dataframe stocks_pivot, by pivoting the stocks_last10 dataframe\n",
    "7. Find difference between adjclose for each pair of consecutive days  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+-------------------+---------+---------+---------+---------+---------+---------+------+\n",
      "|               date|     open|     high|      low|    close|   volume| adjclose|symbol|\n",
      "+-------------------+---------+---------+---------+---------+---------+---------+------+\n",
      "|2000-07-17 00:00:00|  95.4375|     97.5|    92.75|   96.625|3508100.0|74.269199|  XLNX|\n",
      "|2000-07-17 00:00:00|   22.625|    22.75|  22.4375|  22.5625| 201600.0| 13.48614|    ES|\n",
      "|2000-07-17 00:00:00| 6.750002| 6.937503|    6.375|      6.5|1235700.0| 5.241649|   CHK|\n",
      "|2000-07-17 00:00:00|19.812501|  20.1875|19.500001|  20.1875|1434100.0| 3.806147|    NI|\n",
      "|2000-07-17 00:00:00|     30.5|  30.6875|     30.0| 30.03125| 254600.0| 19.81183|   SNA|\n",
      "|2000-07-17 00:00:00|44.749996|45.062498|44.500004|45.000009| 535200.0|17.400773|  FOXA|\n",
      "|2000-07-17 00:00:00|   19.625|   19.625|    19.25|   19.375| 309500.0|13.768835|     R|\n",
      "|2000-07-17 00:00:00|  16.6562|  16.6875|   16.125|    16.25|5507200.0| 1.755466|  ROST|\n",
      "|2000-07-17 00:00:00|    56.25|    57.25|  56.0625|   56.125|7941200.0| 18.31076|    PG|\n",
      "|2000-07-17 00:00:00|54.000326|54.000326|52.500318|53.375325|3725000.0|71.068871|   TYC|\n",
      "|2000-07-17 00:00:00|    58.75|   58.875|  57.8125|     58.0| 182700.0|37.544123|    XL|\n",
      "|2000-07-17 00:00:00|47.500132|47.500132|45.750135|46.343886|4898700.0|17.662922|     F|\n",
      "|2000-07-17 00:00:00|     84.0|     84.5|   82.625|82.671883|2861800.0| 23.88973|   CVX|\n",
      "|2000-07-17 00:00:00|     22.5|    22.75|   22.375|    22.75| 423600.0| 5.942444|   PPL|\n",
      "|2000-07-17 00:00:00|  37.4375|  37.5625|  36.5625|  37.4375| 738800.0|24.832407|   TRV|\n",
      "|2000-07-17 00:00:00|76.874999|80.937504|76.249993|78.250003|6166200.0| 50.37851|     A|\n",
      "|2000-07-17 00:00:00|     26.5|  26.6875|  26.3125|     26.5| 335200.0| 6.240835|   LNT|\n",
      "|2000-07-17 00:00:00|  23.9375|  24.0625|     23.5|  23.9375| 648400.0|  23.9375|   AZO|\n",
      "|2000-07-17 00:00:00|   60.875|  60.9375|    60.25|60.531239|1464800.0|22.017028|   UTX|\n",
      "|2000-07-17 00:00:00|   3.5625| 3.625005| 3.312495| 3.437505| 340000.0|  2.20411|   RRC|\n",
      "+-------------------+---------+---------+---------+---------+---------+---------+------+\n",
      "only showing top 20 rows\n",
      "\n"
     ]
    }
   ],
   "source": [
    "stocks = (spark\n",
    "          .read\n",
    "          .format(\"csv\")\n",
    "          .options(inferSchema = True, header = True)\n",
    "          .load(\"stocks\")\n",
    "          .cache())\n",
    "stocks.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "root\n",
      " |-- date: timestamp (nullable = true)\n",
      " |-- open: double (nullable = true)\n",
      " |-- high: double (nullable = true)\n",
      " |-- low: double (nullable = true)\n",
      " |-- close: double (nullable = true)\n",
      " |-- volume: double (nullable = true)\n",
      " |-- adjclose: double (nullable = true)\n",
      " |-- symbol: string (nullable = true)\n",
      "\n"
     ]
    }
   ],
   "source": [
    "stocks.printSchema()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "root\n",
      " |-- date: date (nullable = true)\n",
      " |-- open: double (nullable = true)\n",
      " |-- high: double (nullable = true)\n",
      " |-- low: double (nullable = true)\n",
      " |-- close: double (nullable = true)\n",
      " |-- volume: double (nullable = true)\n",
      " |-- adjclose: double (nullable = true)\n",
      " |-- symbol: string (nullable = true)\n",
      "\n"
     ]
    }
   ],
   "source": [
    "stocks.withColumn(\"date\", col(\"date\").cast(\"date\")).printSchema()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+----------+---------+---------+---------+---------+---------+---------+------+\n",
      "|      date|     open|     high|      low|    close|   volume| adjclose|symbol|\n",
      "+----------+---------+---------+---------+---------+---------+---------+------+\n",
      "|2000-07-17|  95.4375|     97.5|    92.75|   96.625|3508100.0|74.269199|  XLNX|\n",
      "|2000-07-17|   22.625|    22.75|  22.4375|  22.5625| 201600.0| 13.48614|    ES|\n",
      "|2000-07-17| 6.750002| 6.937503|    6.375|      6.5|1235700.0| 5.241649|   CHK|\n",
      "|2000-07-17|19.812501|  20.1875|19.500001|  20.1875|1434100.0| 3.806147|    NI|\n",
      "|2000-07-17|     30.5|  30.6875|     30.0| 30.03125| 254600.0| 19.81183|   SNA|\n",
      "|2000-07-17|44.749996|45.062498|44.500004|45.000009| 535200.0|17.400773|  FOXA|\n",
      "|2000-07-17|   19.625|   19.625|    19.25|   19.375| 309500.0|13.768835|     R|\n",
      "|2000-07-17|  16.6562|  16.6875|   16.125|    16.25|5507200.0| 1.755466|  ROST|\n",
      "|2000-07-17|    56.25|    57.25|  56.0625|   56.125|7941200.0| 18.31076|    PG|\n",
      "|2000-07-17|54.000326|54.000326|52.500318|53.375325|3725000.0|71.068871|   TYC|\n",
      "|2000-07-17|    58.75|   58.875|  57.8125|     58.0| 182700.0|37.544123|    XL|\n",
      "|2000-07-17|47.500132|47.500132|45.750135|46.343886|4898700.0|17.662922|     F|\n",
      "|2000-07-17|     84.0|     84.5|   82.625|82.671883|2861800.0| 23.88973|   CVX|\n",
      "|2000-07-17|     22.5|    22.75|   22.375|    22.75| 423600.0| 5.942444|   PPL|\n",
      "|2000-07-17|  37.4375|  37.5625|  36.5625|  37.4375| 738800.0|24.832407|   TRV|\n",
      "|2000-07-17|76.874999|80.937504|76.249993|78.250003|6166200.0| 50.37851|     A|\n",
      "|2000-07-17|     26.5|  26.6875|  26.3125|     26.5| 335200.0| 6.240835|   LNT|\n",
      "|2000-07-17|  23.9375|  24.0625|     23.5|  23.9375| 648400.0|  23.9375|   AZO|\n",
      "|2000-07-17|   60.875|  60.9375|    60.25|60.531239|1464800.0|22.017028|   UTX|\n",
      "|2000-07-17|   3.5625| 3.625005| 3.312495| 3.437505| 340000.0|  2.20411|   RRC|\n",
      "+----------+---------+---------+---------+---------+---------+---------+------+\n",
      "only showing top 20 rows\n",
      "\n"
     ]
    }
   ],
   "source": [
    "stocks = stocks.withColumn(\"date\", col(\"date\").cast(\"date\"))\n",
    "stocks.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+----------+-----+-----+----+---------+---------+---------+------+\n",
      "|      date| open| high| low|    close|   volume| adjclose|symbol|\n",
      "+----------+-----+-----+----+---------+---------+---------+------+\n",
      "|2016-08-15|52.02|52.43|52.0|52.040001|1702400.0|52.040001|  XLNX|\n",
      "+----------+-----+-----+----+---------+---------+---------+------+\n",
      "only showing top 1 row\n",
      "\n"
     ]
    }
   ],
   "source": [
    "stocks.orderBy(col(\"date\").desc()).show(1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [],
   "source": [
    "stocks.createOrReplaceTempView(\"stocks\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+----------+------+----------+-------+\n",
      "|      date|symbol|  adjclose|row_num|\n",
      "+----------+------+----------+-------+\n",
      "|2016-08-15|  ALXN|136.070007|      1|\n",
      "|2016-08-12|  ALXN|133.440002|      2|\n",
      "|2016-08-11|  ALXN|133.669998|      3|\n",
      "|2016-08-10|  ALXN|132.669998|      4|\n",
      "|2016-08-09|  ALXN|     135.0|      5|\n",
      "|2016-08-08|  ALXN|136.080002|      6|\n",
      "|2016-08-05|  ALXN|137.110001|      7|\n",
      "|2016-08-04|  ALXN|135.429993|      8|\n",
      "|2016-08-03|  ALXN|137.660004|      9|\n",
      "|2016-08-02|  ALXN|133.610001|     10|\n",
      "|2016-08-01|  ALXN|     133.5|     11|\n",
      "|2016-08-15|   GIS| 71.040001|      1|\n",
      "|2016-08-12|   GIS| 71.169998|      2|\n",
      "|2016-08-11|   GIS|     71.07|      3|\n",
      "|2016-08-10|   GIS| 71.059998|      4|\n",
      "|2016-08-09|   GIS|     70.93|      5|\n",
      "|2016-08-08|   GIS| 70.629997|      6|\n",
      "|2016-08-05|   GIS| 70.550003|      7|\n",
      "|2016-08-04|   GIS| 70.489998|      8|\n",
      "|2016-08-03|   GIS| 69.779999|      9|\n",
      "+----------+------+----------+-------+\n",
      "only showing top 20 rows\n",
      "\n"
     ]
    }
   ],
   "source": [
    "stocks_last10 = sql(\"\"\"\n",
    "select \n",
    "        cast(t1.date as string),\n",
    "        t1.symbol,\n",
    "        t1.adjclose,\n",
    "        t1.row_num\n",
    "    from (select \n",
    "            *, \n",
    "            row_number() over (partition by symbol order by date desc) row_num \n",
    "        from stocks) t1 where t1.row_num <= 11\n",
    "\"\"\")\n",
    "\n",
    "stocks_last10.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>symbol</th>\n",
       "      <th>2016-08-01</th>\n",
       "      <th>2016-08-02</th>\n",
       "      <th>2016-08-03</th>\n",
       "      <th>2016-08-04</th>\n",
       "      <th>2016-08-05</th>\n",
       "      <th>2016-08-08</th>\n",
       "      <th>2016-08-09</th>\n",
       "      <th>2016-08-10</th>\n",
       "      <th>2016-08-11</th>\n",
       "      <th>2016-08-12</th>\n",
       "      <th>2016-08-15</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>ALXN</td>\n",
       "      <td>133.500000</td>\n",
       "      <td>133.610001</td>\n",
       "      <td>137.660004</td>\n",
       "      <td>135.429993</td>\n",
       "      <td>137.110001</td>\n",
       "      <td>136.080002</td>\n",
       "      <td>135.000000</td>\n",
       "      <td>132.669998</td>\n",
       "      <td>133.669998</td>\n",
       "      <td>133.440002</td>\n",
       "      <td>136.070007</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>GIS</td>\n",
       "      <td>71.629997</td>\n",
       "      <td>70.970001</td>\n",
       "      <td>69.779999</td>\n",
       "      <td>70.489998</td>\n",
       "      <td>70.550003</td>\n",
       "      <td>70.629997</td>\n",
       "      <td>70.930000</td>\n",
       "      <td>71.059998</td>\n",
       "      <td>71.070000</td>\n",
       "      <td>71.169998</td>\n",
       "      <td>71.040001</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>K</td>\n",
       "      <td>82.370003</td>\n",
       "      <td>81.870003</td>\n",
       "      <td>81.080002</td>\n",
       "      <td>82.419998</td>\n",
       "      <td>82.709999</td>\n",
       "      <td>82.919998</td>\n",
       "      <td>82.980003</td>\n",
       "      <td>82.959999</td>\n",
       "      <td>83.459999</td>\n",
       "      <td>83.529999</td>\n",
       "      <td>83.389999</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>LEN</td>\n",
       "      <td>46.750000</td>\n",
       "      <td>45.529999</td>\n",
       "      <td>45.740002</td>\n",
       "      <td>45.810001</td>\n",
       "      <td>46.799999</td>\n",
       "      <td>46.619999</td>\n",
       "      <td>46.830002</td>\n",
       "      <td>46.799999</td>\n",
       "      <td>47.160000</td>\n",
       "      <td>46.750000</td>\n",
       "      <td>47.299999</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>SPGI</td>\n",
       "      <td>121.220001</td>\n",
       "      <td>120.110001</td>\n",
       "      <td>120.510002</td>\n",
       "      <td>119.620003</td>\n",
       "      <td>120.900002</td>\n",
       "      <td>120.160004</td>\n",
       "      <td>120.580002</td>\n",
       "      <td>119.830002</td>\n",
       "      <td>120.410004</td>\n",
       "      <td>119.910004</td>\n",
       "      <td>120.260002</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>AIV</td>\n",
       "      <td>46.060001</td>\n",
       "      <td>45.169998</td>\n",
       "      <td>45.049999</td>\n",
       "      <td>44.880001</td>\n",
       "      <td>45.360001</td>\n",
       "      <td>45.880001</td>\n",
       "      <td>45.980000</td>\n",
       "      <td>45.400002</td>\n",
       "      <td>44.869999</td>\n",
       "      <td>45.049999</td>\n",
       "      <td>45.540001</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>AVY</td>\n",
       "      <td>78.150002</td>\n",
       "      <td>77.849998</td>\n",
       "      <td>77.239998</td>\n",
       "      <td>77.389999</td>\n",
       "      <td>78.459999</td>\n",
       "      <td>78.720001</td>\n",
       "      <td>78.730003</td>\n",
       "      <td>78.690002</td>\n",
       "      <td>78.300003</td>\n",
       "      <td>78.190002</td>\n",
       "      <td>78.169998</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>MMM</td>\n",
       "      <td>178.289993</td>\n",
       "      <td>178.360001</td>\n",
       "      <td>178.380005</td>\n",
       "      <td>177.809998</td>\n",
       "      <td>178.570007</td>\n",
       "      <td>178.580002</td>\n",
       "      <td>178.389999</td>\n",
       "      <td>178.820007</td>\n",
       "      <td>181.000000</td>\n",
       "      <td>180.270004</td>\n",
       "      <td>180.559998</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>PKI</td>\n",
       "      <td>56.689999</td>\n",
       "      <td>55.910000</td>\n",
       "      <td>55.750000</td>\n",
       "      <td>55.720001</td>\n",
       "      <td>54.849998</td>\n",
       "      <td>54.590000</td>\n",
       "      <td>54.720001</td>\n",
       "      <td>54.540001</td>\n",
       "      <td>54.720001</td>\n",
       "      <td>54.410000</td>\n",
       "      <td>54.610001</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>PPG</td>\n",
       "      <td>104.269079</td>\n",
       "      <td>103.292837</td>\n",
       "      <td>103.930385</td>\n",
       "      <td>103.701263</td>\n",
       "      <td>104.030001</td>\n",
       "      <td>104.480003</td>\n",
       "      <td>104.370003</td>\n",
       "      <td>104.690002</td>\n",
       "      <td>104.739998</td>\n",
       "      <td>103.989998</td>\n",
       "      <td>104.459999</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  symbol  2016-08-01  2016-08-02  2016-08-03  2016-08-04  2016-08-05  \\\n",
       "0   ALXN  133.500000  133.610001  137.660004  135.429993  137.110001   \n",
       "1    GIS   71.629997   70.970001   69.779999   70.489998   70.550003   \n",
       "2      K   82.370003   81.870003   81.080002   82.419998   82.709999   \n",
       "3    LEN   46.750000   45.529999   45.740002   45.810001   46.799999   \n",
       "4   SPGI  121.220001  120.110001  120.510002  119.620003  120.900002   \n",
       "5    AIV   46.060001   45.169998   45.049999   44.880001   45.360001   \n",
       "6    AVY   78.150002   77.849998   77.239998   77.389999   78.459999   \n",
       "7    MMM  178.289993  178.360001  178.380005  177.809998  178.570007   \n",
       "8    PKI   56.689999   55.910000   55.750000   55.720001   54.849998   \n",
       "9    PPG  104.269079  103.292837  103.930385  103.701263  104.030001   \n",
       "\n",
       "   2016-08-08  2016-08-09  2016-08-10  2016-08-11  2016-08-12  2016-08-15  \n",
       "0  136.080002  135.000000  132.669998  133.669998  133.440002  136.070007  \n",
       "1   70.629997   70.930000   71.059998   71.070000   71.169998   71.040001  \n",
       "2   82.919998   82.980003   82.959999   83.459999   83.529999   83.389999  \n",
       "3   46.619999   46.830002   46.799999   47.160000   46.750000   47.299999  \n",
       "4  120.160004  120.580002  119.830002  120.410004  119.910004  120.260002  \n",
       "5   45.880001   45.980000   45.400002   44.869999   45.049999   45.540001  \n",
       "6   78.720001   78.730003   78.690002   78.300003   78.190002   78.169998  \n",
       "7  178.580002  178.389999  178.820007  181.000000  180.270004  180.559998  \n",
       "8   54.590000   54.720001   54.540001   54.720001   54.410000   54.610001  \n",
       "9  104.480003  104.370003  104.690002  104.739998  103.989998  104.459999  "
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "stocks_pivot = stocks_last10.groupby(\"symbol\").pivot(\"date\").agg(max(col(\"adjclose\")))\n",
    "stocks_pivot.limit(10).toPandas()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['symbol', 'v0', 'v1', 'v2', 'v3', 'v4', 'v5', 'v6', 'v7', 'v8', 'v9', 'v10']"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "columns = [\"v\" + str(i) for i in range(len(stocks_pivot.columns) - 1)]\n",
    "columns.insert(0, \"symbol\")\n",
    "columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>symbol</th>\n",
       "      <th>v0</th>\n",
       "      <th>v1</th>\n",
       "      <th>v2</th>\n",
       "      <th>v3</th>\n",
       "      <th>v4</th>\n",
       "      <th>v5</th>\n",
       "      <th>v6</th>\n",
       "      <th>v7</th>\n",
       "      <th>v8</th>\n",
       "      <th>v9</th>\n",
       "      <th>v10</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>ALXN</td>\n",
       "      <td>133.500000</td>\n",
       "      <td>133.610001</td>\n",
       "      <td>137.660004</td>\n",
       "      <td>135.429993</td>\n",
       "      <td>137.110001</td>\n",
       "      <td>136.080002</td>\n",
       "      <td>135.000000</td>\n",
       "      <td>132.669998</td>\n",
       "      <td>133.669998</td>\n",
       "      <td>133.440002</td>\n",
       "      <td>136.070007</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>GIS</td>\n",
       "      <td>71.629997</td>\n",
       "      <td>70.970001</td>\n",
       "      <td>69.779999</td>\n",
       "      <td>70.489998</td>\n",
       "      <td>70.550003</td>\n",
       "      <td>70.629997</td>\n",
       "      <td>70.930000</td>\n",
       "      <td>71.059998</td>\n",
       "      <td>71.070000</td>\n",
       "      <td>71.169998</td>\n",
       "      <td>71.040001</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>K</td>\n",
       "      <td>82.370003</td>\n",
       "      <td>81.870003</td>\n",
       "      <td>81.080002</td>\n",
       "      <td>82.419998</td>\n",
       "      <td>82.709999</td>\n",
       "      <td>82.919998</td>\n",
       "      <td>82.980003</td>\n",
       "      <td>82.959999</td>\n",
       "      <td>83.459999</td>\n",
       "      <td>83.529999</td>\n",
       "      <td>83.389999</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>LEN</td>\n",
       "      <td>46.750000</td>\n",
       "      <td>45.529999</td>\n",
       "      <td>45.740002</td>\n",
       "      <td>45.810001</td>\n",
       "      <td>46.799999</td>\n",
       "      <td>46.619999</td>\n",
       "      <td>46.830002</td>\n",
       "      <td>46.799999</td>\n",
       "      <td>47.160000</td>\n",
       "      <td>46.750000</td>\n",
       "      <td>47.299999</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>SPGI</td>\n",
       "      <td>121.220001</td>\n",
       "      <td>120.110001</td>\n",
       "      <td>120.510002</td>\n",
       "      <td>119.620003</td>\n",
       "      <td>120.900002</td>\n",
       "      <td>120.160004</td>\n",
       "      <td>120.580002</td>\n",
       "      <td>119.830002</td>\n",
       "      <td>120.410004</td>\n",
       "      <td>119.910004</td>\n",
       "      <td>120.260002</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>AIV</td>\n",
       "      <td>46.060001</td>\n",
       "      <td>45.169998</td>\n",
       "      <td>45.049999</td>\n",
       "      <td>44.880001</td>\n",
       "      <td>45.360001</td>\n",
       "      <td>45.880001</td>\n",
       "      <td>45.980000</td>\n",
       "      <td>45.400002</td>\n",
       "      <td>44.869999</td>\n",
       "      <td>45.049999</td>\n",
       "      <td>45.540001</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>AVY</td>\n",
       "      <td>78.150002</td>\n",
       "      <td>77.849998</td>\n",
       "      <td>77.239998</td>\n",
       "      <td>77.389999</td>\n",
       "      <td>78.459999</td>\n",
       "      <td>78.720001</td>\n",
       "      <td>78.730003</td>\n",
       "      <td>78.690002</td>\n",
       "      <td>78.300003</td>\n",
       "      <td>78.190002</td>\n",
       "      <td>78.169998</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>MMM</td>\n",
       "      <td>178.289993</td>\n",
       "      <td>178.360001</td>\n",
       "      <td>178.380005</td>\n",
       "      <td>177.809998</td>\n",
       "      <td>178.570007</td>\n",
       "      <td>178.580002</td>\n",
       "      <td>178.389999</td>\n",
       "      <td>178.820007</td>\n",
       "      <td>181.000000</td>\n",
       "      <td>180.270004</td>\n",
       "      <td>180.559998</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>PKI</td>\n",
       "      <td>56.689999</td>\n",
       "      <td>55.910000</td>\n",
       "      <td>55.750000</td>\n",
       "      <td>55.720001</td>\n",
       "      <td>54.849998</td>\n",
       "      <td>54.590000</td>\n",
       "      <td>54.720001</td>\n",
       "      <td>54.540001</td>\n",
       "      <td>54.720001</td>\n",
       "      <td>54.410000</td>\n",
       "      <td>54.610001</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>PPG</td>\n",
       "      <td>104.269079</td>\n",
       "      <td>103.292837</td>\n",
       "      <td>103.930385</td>\n",
       "      <td>103.701263</td>\n",
       "      <td>104.030001</td>\n",
       "      <td>104.480003</td>\n",
       "      <td>104.370003</td>\n",
       "      <td>104.690002</td>\n",
       "      <td>104.739998</td>\n",
       "      <td>103.989998</td>\n",
       "      <td>104.459999</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  symbol          v0          v1          v2          v3          v4  \\\n",
       "0   ALXN  133.500000  133.610001  137.660004  135.429993  137.110001   \n",
       "1    GIS   71.629997   70.970001   69.779999   70.489998   70.550003   \n",
       "2      K   82.370003   81.870003   81.080002   82.419998   82.709999   \n",
       "3    LEN   46.750000   45.529999   45.740002   45.810001   46.799999   \n",
       "4   SPGI  121.220001  120.110001  120.510002  119.620003  120.900002   \n",
       "5    AIV   46.060001   45.169998   45.049999   44.880001   45.360001   \n",
       "6    AVY   78.150002   77.849998   77.239998   77.389999   78.459999   \n",
       "7    MMM  178.289993  178.360001  178.380005  177.809998  178.570007   \n",
       "8    PKI   56.689999   55.910000   55.750000   55.720001   54.849998   \n",
       "9    PPG  104.269079  103.292837  103.930385  103.701263  104.030001   \n",
       "\n",
       "           v5          v6          v7          v8          v9         v10  \n",
       "0  136.080002  135.000000  132.669998  133.669998  133.440002  136.070007  \n",
       "1   70.629997   70.930000   71.059998   71.070000   71.169998   71.040001  \n",
       "2   82.919998   82.980003   82.959999   83.459999   83.529999   83.389999  \n",
       "3   46.619999   46.830002   46.799999   47.160000   46.750000   47.299999  \n",
       "4  120.160004  120.580002  119.830002  120.410004  119.910004  120.260002  \n",
       "5   45.880001   45.980000   45.400002   44.869999   45.049999   45.540001  \n",
       "6   78.720001   78.730003   78.690002   78.300003   78.190002   78.169998  \n",
       "7  178.580002  178.389999  178.820007  181.000000  180.270004  180.559998  \n",
       "8   54.590000   54.720001   54.540001   54.720001   54.410000   54.610001  \n",
       "9  104.480003  104.370003  104.690002  104.739998  103.989998  104.459999  "
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "stocks_pivot = stocks_pivot.toDF(*columns)\n",
    "stocks_pivot.limit(10).toPandas()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>symbol</th>\n",
       "      <th>v0</th>\n",
       "      <th>v1</th>\n",
       "      <th>v2</th>\n",
       "      <th>v3</th>\n",
       "      <th>v4</th>\n",
       "      <th>v5</th>\n",
       "      <th>v6</th>\n",
       "      <th>v7</th>\n",
       "      <th>v8</th>\n",
       "      <th>v9</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>ALXN</td>\n",
       "      <td>-0.110001</td>\n",
       "      <td>-4.050003</td>\n",
       "      <td>2.230011</td>\n",
       "      <td>-1.680008</td>\n",
       "      <td>1.029999</td>\n",
       "      <td>1.080002</td>\n",
       "      <td>2.330002</td>\n",
       "      <td>-1.000000</td>\n",
       "      <td>0.229996</td>\n",
       "      <td>-2.630005</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>GIS</td>\n",
       "      <td>0.659996</td>\n",
       "      <td>1.190002</td>\n",
       "      <td>-0.709999</td>\n",
       "      <td>-0.060005</td>\n",
       "      <td>-0.079994</td>\n",
       "      <td>-0.300003</td>\n",
       "      <td>-0.129998</td>\n",
       "      <td>-0.010002</td>\n",
       "      <td>-0.099998</td>\n",
       "      <td>0.129997</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>K</td>\n",
       "      <td>0.500000</td>\n",
       "      <td>0.790001</td>\n",
       "      <td>-1.339996</td>\n",
       "      <td>-0.290001</td>\n",
       "      <td>-0.209999</td>\n",
       "      <td>-0.060005</td>\n",
       "      <td>0.020004</td>\n",
       "      <td>-0.500000</td>\n",
       "      <td>-0.070000</td>\n",
       "      <td>0.140000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>LEN</td>\n",
       "      <td>1.220001</td>\n",
       "      <td>-0.210003</td>\n",
       "      <td>-0.069999</td>\n",
       "      <td>-0.989998</td>\n",
       "      <td>0.180000</td>\n",
       "      <td>-0.210003</td>\n",
       "      <td>0.030003</td>\n",
       "      <td>-0.360001</td>\n",
       "      <td>0.410000</td>\n",
       "      <td>-0.549999</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>SPGI</td>\n",
       "      <td>1.110000</td>\n",
       "      <td>-0.400001</td>\n",
       "      <td>0.889999</td>\n",
       "      <td>-1.279999</td>\n",
       "      <td>0.739998</td>\n",
       "      <td>-0.419998</td>\n",
       "      <td>0.750000</td>\n",
       "      <td>-0.580002</td>\n",
       "      <td>0.500000</td>\n",
       "      <td>-0.349998</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>AIV</td>\n",
       "      <td>0.890003</td>\n",
       "      <td>0.119999</td>\n",
       "      <td>0.169998</td>\n",
       "      <td>-0.480000</td>\n",
       "      <td>-0.520000</td>\n",
       "      <td>-0.099999</td>\n",
       "      <td>0.579998</td>\n",
       "      <td>0.530003</td>\n",
       "      <td>-0.180000</td>\n",
       "      <td>-0.490002</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>AVY</td>\n",
       "      <td>0.300004</td>\n",
       "      <td>0.610000</td>\n",
       "      <td>-0.150001</td>\n",
       "      <td>-1.070000</td>\n",
       "      <td>-0.260002</td>\n",
       "      <td>-0.010002</td>\n",
       "      <td>0.040001</td>\n",
       "      <td>0.389999</td>\n",
       "      <td>0.110001</td>\n",
       "      <td>0.020004</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>MMM</td>\n",
       "      <td>-0.070008</td>\n",
       "      <td>-0.020004</td>\n",
       "      <td>0.570007</td>\n",
       "      <td>-0.760009</td>\n",
       "      <td>-0.009995</td>\n",
       "      <td>0.190003</td>\n",
       "      <td>-0.430008</td>\n",
       "      <td>-2.179993</td>\n",
       "      <td>0.729996</td>\n",
       "      <td>-0.289994</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>PKI</td>\n",
       "      <td>0.779999</td>\n",
       "      <td>0.160000</td>\n",
       "      <td>0.029999</td>\n",
       "      <td>0.870003</td>\n",
       "      <td>0.259998</td>\n",
       "      <td>-0.130001</td>\n",
       "      <td>0.180000</td>\n",
       "      <td>-0.180000</td>\n",
       "      <td>0.310001</td>\n",
       "      <td>-0.200001</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>PPG</td>\n",
       "      <td>0.976242</td>\n",
       "      <td>-0.637548</td>\n",
       "      <td>0.229122</td>\n",
       "      <td>-0.328738</td>\n",
       "      <td>-0.450002</td>\n",
       "      <td>0.110000</td>\n",
       "      <td>-0.319999</td>\n",
       "      <td>-0.049996</td>\n",
       "      <td>0.750000</td>\n",
       "      <td>-0.470001</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  symbol        v0        v1        v2        v3        v4        v5  \\\n",
       "0   ALXN -0.110001 -4.050003  2.230011 -1.680008  1.029999  1.080002   \n",
       "1    GIS  0.659996  1.190002 -0.709999 -0.060005 -0.079994 -0.300003   \n",
       "2      K  0.500000  0.790001 -1.339996 -0.290001 -0.209999 -0.060005   \n",
       "3    LEN  1.220001 -0.210003 -0.069999 -0.989998  0.180000 -0.210003   \n",
       "4   SPGI  1.110000 -0.400001  0.889999 -1.279999  0.739998 -0.419998   \n",
       "5    AIV  0.890003  0.119999  0.169998 -0.480000 -0.520000 -0.099999   \n",
       "6    AVY  0.300004  0.610000 -0.150001 -1.070000 -0.260002 -0.010002   \n",
       "7    MMM -0.070008 -0.020004  0.570007 -0.760009 -0.009995  0.190003   \n",
       "8    PKI  0.779999  0.160000  0.029999  0.870003  0.259998 -0.130001   \n",
       "9    PPG  0.976242 -0.637548  0.229122 -0.328738 -0.450002  0.110000   \n",
       "\n",
       "         v6        v7        v8        v9  \n",
       "0  2.330002 -1.000000  0.229996 -2.630005  \n",
       "1 -0.129998 -0.010002 -0.099998  0.129997  \n",
       "2  0.020004 -0.500000 -0.070000  0.140000  \n",
       "3  0.030003 -0.360001  0.410000 -0.549999  \n",
       "4  0.750000 -0.580002  0.500000 -0.349998  \n",
       "5  0.579998  0.530003 -0.180000 -0.490002  \n",
       "6  0.040001  0.389999  0.110001  0.020004  \n",
       "7 -0.430008 -2.179993  0.729996 -0.289994  \n",
       "8  0.180000 -0.180000  0.310001 -0.200001  \n",
       "9 -0.319999 -0.049996  0.750000 -0.470001  "
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "stocks_diff = stocks_pivot\n",
    "for i in range(1, len(stocks_diff.columns) - 1):\n",
    "    stocks_diff = stocks_diff.withColumn(columns[i], col(columns[i]) - col(columns[i+1]))\n",
    "\n",
    "stocks_diff = stocks_diff.drop(stocks_diff.columns[-1])\n",
    "stocks_diff.limit(10).toPandas()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Exercise 3: Read data from mysql \n",
    "\n",
    "1. Create a dataframe - orders in Spark based on orders table in retail_db in mysql\n",
    "2. Save the orders as parquet file in HDFS\n",
    "3. Write a query joining customers table in mysql and orders parquet file in HDFS to find customers with most number of completed orders.\n",
    "4. Save the orders dataframe as hive table. Verify that orders table is accessible in hive as well.\n",
    "5. Delete the orders table from hive."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Create a dataframe - orders in Spark based on orders table in retail_db in mysql"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+--------+-------------------+-----------------+---------------+\n",
      "|order_id|         order_date|order_customer_id|   order_status|\n",
      "+--------+-------------------+-----------------+---------------+\n",
      "|       1|2013-07-25 00:00:00|            11599|         CLOSED|\n",
      "|       2|2013-07-25 00:00:00|              256|PENDING_PAYMENT|\n",
      "|       3|2013-07-25 00:00:00|            12111|       COMPLETE|\n",
      "|       4|2013-07-25 00:00:00|             8827|         CLOSED|\n",
      "|       5|2013-07-25 00:00:00|            11318|       COMPLETE|\n",
      "|       6|2013-07-25 00:00:00|             7130|       COMPLETE|\n",
      "|       7|2013-07-25 00:00:00|             4530|       COMPLETE|\n",
      "|       8|2013-07-25 00:00:00|             2911|     PROCESSING|\n",
      "|       9|2013-07-25 00:00:00|             5657|PENDING_PAYMENT|\n",
      "|      10|2013-07-25 00:00:00|             5648|PENDING_PAYMENT|\n",
      "|      11|2013-07-25 00:00:00|              918| PAYMENT_REVIEW|\n",
      "|      12|2013-07-25 00:00:00|             1837|         CLOSED|\n",
      "|      13|2013-07-25 00:00:00|             9149|PENDING_PAYMENT|\n",
      "|      14|2013-07-25 00:00:00|             9842|     PROCESSING|\n",
      "|      15|2013-07-25 00:00:00|             2568|       COMPLETE|\n",
      "|      16|2013-07-25 00:00:00|             7276|PENDING_PAYMENT|\n",
      "|      17|2013-07-25 00:00:00|             2667|       COMPLETE|\n",
      "|      18|2013-07-25 00:00:00|             1205|         CLOSED|\n",
      "|      19|2013-07-25 00:00:00|             9488|PENDING_PAYMENT|\n",
      "|      20|2013-07-25 00:00:00|             9198|     PROCESSING|\n",
      "+--------+-------------------+-----------------+---------------+\n",
      "only showing top 20 rows\n",
      "\n"
     ]
    }
   ],
   "source": [
    "orders = (spark\n",
    "             .read\n",
    "             .format(\"jdbc\")\n",
    "             .option(\"url\", \"jdbc:mysql://localhost/retail_db\")\n",
    "             .option(\"driver\", \"com.mysql.jdbc.Driver\")\n",
    "             .option(\"dbtable\", \"orders\")\n",
    "             .option(\"user\", \"root\")\n",
    "             .option(\"password\", \"cloudera\")\n",
    "             .load())\n",
    "orders.show()         "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Save the orders as parquet file in HDFS"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [],
   "source": [
    "orders.write.format(\"parquet\").mode(\"overwrite\").save(\"orders\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Create a dataframe customers, based on the customers table in retail_db in mysql database."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+-----------+--------------+--------------+--------------+-----------------+--------------------+-------------+--------------+----------------+\n",
      "|customer_id|customer_fname|customer_lname|customer_email|customer_password|     customer_street|customer_city|customer_state|customer_zipcode|\n",
      "+-----------+--------------+--------------+--------------+-----------------+--------------------+-------------+--------------+----------------+\n",
      "|          1|       Richard|     Hernandez|     XXXXXXXXX|        XXXXXXXXX|  6303 Heather Plaza|  Brownsville|            TX|           78521|\n",
      "|          2|          Mary|       Barrett|     XXXXXXXXX|        XXXXXXXXX|9526 Noble Embers...|    Littleton|            CO|           80126|\n",
      "|          3|           Ann|         Smith|     XXXXXXXXX|        XXXXXXXXX|3422 Blue Pioneer...|       Caguas|            PR|           00725|\n",
      "|          4|          Mary|         Jones|     XXXXXXXXX|        XXXXXXXXX|  8324 Little Common|   San Marcos|            CA|           92069|\n",
      "|          5|        Robert|        Hudson|     XXXXXXXXX|        XXXXXXXXX|10 Crystal River ...|       Caguas|            PR|           00725|\n",
      "+-----------+--------------+--------------+--------------+-----------------+--------------------+-------------+--------------+----------------+\n",
      "only showing top 5 rows\n",
      "\n"
     ]
    }
   ],
   "source": [
    "customers = (spark\n",
    "             .read\n",
    "             .format(\"jdbc\")\n",
    "             .option(\"url\", \"jdbc:mysql://localhost/retail_db\")\n",
    "             .option(\"driver\", \"com.mysql.jdbc.Driver\")\n",
    "             .option(\"dbtable\", \"customers\")\n",
    "             .option(\"user\", \"root\")\n",
    "             .option(\"password\", \"cloudera\")\n",
    "             .load())\n",
    "customers.show(5)    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Create an orders based on the orders parquet file. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+--------+-------------------+-----------------+---------------+\n",
      "|order_id|         order_date|order_customer_id|   order_status|\n",
      "+--------+-------------------+-----------------+---------------+\n",
      "|       1|2013-07-25 00:00:00|            11599|         CLOSED|\n",
      "|       2|2013-07-25 00:00:00|              256|PENDING_PAYMENT|\n",
      "|       3|2013-07-25 00:00:00|            12111|       COMPLETE|\n",
      "|       4|2013-07-25 00:00:00|             8827|         CLOSED|\n",
      "|       5|2013-07-25 00:00:00|            11318|       COMPLETE|\n",
      "|       6|2013-07-25 00:00:00|             7130|       COMPLETE|\n",
      "+--------+-------------------+-----------------+---------------+\n",
      "only showing top 6 rows\n",
      "\n"
     ]
    }
   ],
   "source": [
    "orders = spark.read.load(\"orders\")\n",
    "orders.show(6)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Write a query joining customers table in mysql and orders parquet file in HDFS to find most number of complet orders."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+-----------------+--------------+--------------+-----+\n",
      "|order_customer_id|customer_fname|customer_lname|count|\n",
      "+-----------------+--------------+--------------+-----+\n",
      "|             9337|          Mary|         Smith|   10|\n",
      "|             7802|          Mary|       Acevedo|    9|\n",
      "|             3710|        Ashley|         Smith|    9|\n",
      "|              749|         Jesse|      Matthews|    9|\n",
      "|             5186|         Jason|      Robinson|    8|\n",
      "|              221|          Mary|           Cox|    8|\n",
      "|             2469|         Shawn|         Smith|    8|\n",
      "|            11061|        Joseph|       Webster|    8|\n",
      "|             5283|         Jacob|      Guerrero|    8|\n",
      "|             7910|          Mary|       Camacho|    8|\n",
      "+-----------------+--------------+--------------+-----+\n",
      "only showing top 10 rows\n",
      "\n"
     ]
    }
   ],
   "source": [
    "(orders.alias(\"t1\")\n",
    ".join(customers.alias(\"t2\"), col(\"t1.order_customer_id\") == col(\"t2.customer_id\"))\n",
    ".filter(\"t1.order_status == 'COMPLETE'\")\n",
    ".groupby(\"order_customer_id\", \"customer_fname\", \"customer_lname\")\n",
    ".count()\n",
    ".orderBy(col(\"count\").desc())\n",
    ".show(10))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [],
   "source": [
    "orders.write.mode(\"overwrite\").saveAsTable(\"orders\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Verify the table in Hive. See whether orders table shows up as a permanent table. You can use use describe formatted <table> command to see what type of hive file it is."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+--------+---------+-----------+\n",
      "|database|tableName|isTemporary|\n",
      "+--------+---------+-----------+\n",
      "| default|   movies|      false|\n",
      "| default|   orders|      false|\n",
      "| default|  weblogs|      false|\n",
      "|        |   movies|       true|\n",
      "|        |  ratings|       true|\n",
      "|        |   stocks|       true|\n",
      "+--------+---------+-----------+\n",
      "\n"
     ]
    }
   ],
   "source": [
    "sql(\"show tables\").show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+--------+-------------------+-----------------+---------------+\n",
      "|order_id|         order_date|order_customer_id|   order_status|\n",
      "+--------+-------------------+-----------------+---------------+\n",
      "|       1|2013-07-25 00:00:00|            11599|         CLOSED|\n",
      "|       2|2013-07-25 00:00:00|              256|PENDING_PAYMENT|\n",
      "|       3|2013-07-25 00:00:00|            12111|       COMPLETE|\n",
      "|       4|2013-07-25 00:00:00|             8827|         CLOSED|\n",
      "|       5|2013-07-25 00:00:00|            11318|       COMPLETE|\n",
      "|       6|2013-07-25 00:00:00|             7130|       COMPLETE|\n",
      "|       7|2013-07-25 00:00:00|             4530|       COMPLETE|\n",
      "|       8|2013-07-25 00:00:00|             2911|     PROCESSING|\n",
      "|       9|2013-07-25 00:00:00|             5657|PENDING_PAYMENT|\n",
      "|      10|2013-07-25 00:00:00|             5648|PENDING_PAYMENT|\n",
      "+--------+-------------------+-----------------+---------------+\n",
      "only showing top 10 rows\n",
      "\n"
     ]
    }
   ],
   "source": [
    "spark.table(\"orders\").show(10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>col_name</th>\n",
       "      <th>data_type</th>\n",
       "      <th>comment</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>order_id</td>\n",
       "      <td>int</td>\n",
       "      <td>None</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>order_date</td>\n",
       "      <td>timestamp</td>\n",
       "      <td>None</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>order_customer_id</td>\n",
       "      <td>int</td>\n",
       "      <td>None</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>order_status</td>\n",
       "      <td>string</td>\n",
       "      <td>None</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td># Detailed Table Information</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>Database</td>\n",
       "      <td>default</td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>Table</td>\n",
       "      <td>orders</td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>Owner</td>\n",
       "      <td>cloudera</td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>Created Time</td>\n",
       "      <td>Tue Apr 24 03:36:17 PDT 2018</td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>Last Access</td>\n",
       "      <td>Wed Dec 31 16:00:00 PST 1969</td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>Created By</td>\n",
       "      <td>Spark 2.3.0</td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>Type</td>\n",
       "      <td>MANAGED</td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>Provider</td>\n",
       "      <td>parquet</td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>Table Properties</td>\n",
       "      <td>[transient_lastDdlTime=1524566177]</td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>Statistics</td>\n",
       "      <td>487358 bytes</td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>Location</td>\n",
       "      <td>hdfs://quickstart.cloudera:8020/user/hive/ware...</td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>Serde Library</td>\n",
       "      <td>org.apache.hadoop.hive.ql.io.parquet.serde.Par...</td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18</th>\n",
       "      <td>InputFormat</td>\n",
       "      <td>org.apache.hadoop.hive.ql.io.parquet.MapredPar...</td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19</th>\n",
       "      <td>OutputFormat</td>\n",
       "      <td>org.apache.hadoop.hive.ql.io.parquet.MapredPar...</td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20</th>\n",
       "      <td>Storage Properties</td>\n",
       "      <td>[serialization.format=1]</td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                        col_name  \\\n",
       "0                       order_id   \n",
       "1                     order_date   \n",
       "2              order_customer_id   \n",
       "3                   order_status   \n",
       "4                                  \n",
       "5   # Detailed Table Information   \n",
       "6                       Database   \n",
       "7                          Table   \n",
       "8                          Owner   \n",
       "9                   Created Time   \n",
       "10                   Last Access   \n",
       "11                    Created By   \n",
       "12                          Type   \n",
       "13                      Provider   \n",
       "14              Table Properties   \n",
       "15                    Statistics   \n",
       "16                      Location   \n",
       "17                 Serde Library   \n",
       "18                   InputFormat   \n",
       "19                  OutputFormat   \n",
       "20            Storage Properties   \n",
       "\n",
       "                                            data_type comment  \n",
       "0                                                 int    None  \n",
       "1                                           timestamp    None  \n",
       "2                                                 int    None  \n",
       "3                                              string    None  \n",
       "4                                                              \n",
       "5                                                              \n",
       "6                                             default          \n",
       "7                                              orders          \n",
       "8                                            cloudera          \n",
       "9                        Tue Apr 24 03:36:17 PDT 2018          \n",
       "10                       Wed Dec 31 16:00:00 PST 1969          \n",
       "11                                        Spark 2.3.0          \n",
       "12                                            MANAGED          \n",
       "13                                            parquet          \n",
       "14                 [transient_lastDdlTime=1524566177]          \n",
       "15                                       487358 bytes          \n",
       "16  hdfs://quickstart.cloudera:8020/user/hive/ware...          \n",
       "17  org.apache.hadoop.hive.ql.io.parquet.serde.Par...          \n",
       "18  org.apache.hadoop.hive.ql.io.parquet.MapredPar...          \n",
       "19  org.apache.hadoop.hive.ql.io.parquet.MapredPar...          \n",
       "20                           [serialization.format=1]          "
      ]
     },
     "execution_count": 42,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sql(\"describe formatted orders\").toPandas()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Drop the table from hive"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "++\n",
      "||\n",
      "++\n",
      "++\n",
      "\n"
     ]
    }
   ],
   "source": [
    "sql(\"drop table orders\").show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Exercise 4: Use SFPD (San Francisco Police Department) Dataset for the following tasks\n",
    "\n",
    "\n",
    "Filename: Crime_Incidents.csv.gz. Create a directory in your file system called called sfpd in your home path and put this file into the directory. \n",
    "\n",
    "Tasks:\n",
    "1. Create an dataframe with crime incident data.\n",
    "3. Show the first 10 values of an dataframe\n",
    "4. Check number of partitions of the dataframe\n",
    "5. Find the number of incident records in the dataframe\n",
    "6. Find the categories of incidents\n",
    "7. Find total number of categories\n",
    "8. What are the total number of incidents in each category\n",
    "9. Find out on which day each of incidents have occurred most. \n",
    "10. [Optional] Plot the frequency for each category of events\n",
    "11. Createa a UDF to parse date field to conver it into a date type field"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "root\n",
      " |-- IncidntNum: integer (nullable = true)\n",
      " |-- Category: string (nullable = true)\n",
      " |-- Descript: string (nullable = true)\n",
      " |-- DayOfWeek: string (nullable = true)\n",
      " |-- Date: string (nullable = true)\n",
      " |-- Time: string (nullable = true)\n",
      " |-- PdDistrict: string (nullable = true)\n",
      " |-- Resolution: string (nullable = true)\n",
      " |-- Address: string (nullable = true)\n",
      " |-- X: double (nullable = true)\n",
      " |-- Y: double (nullable = true)\n",
      " |-- Location: string (nullable = true)\n",
      "\n"
     ]
    }
   ],
   "source": [
    "sfpd = (spark\n",
    "        .read\n",
    "        .format(\"csv\")\n",
    "        .option(\"header\", True)\n",
    "        .option(\"inferSchema\", True)\n",
    "        .load(\"sfpd\")\n",
    "       )\n",
    "sfpd.printSchema()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>IncidntNum</th>\n",
       "      <th>Category</th>\n",
       "      <th>Descript</th>\n",
       "      <th>DayOfWeek</th>\n",
       "      <th>Date</th>\n",
       "      <th>Time</th>\n",
       "      <th>PdDistrict</th>\n",
       "      <th>Resolution</th>\n",
       "      <th>Address</th>\n",
       "      <th>X</th>\n",
       "      <th>Y</th>\n",
       "      <th>Location</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>50436712</td>\n",
       "      <td>ASSAULT</td>\n",
       "      <td>BATTERY</td>\n",
       "      <td>Wednesday</td>\n",
       "      <td>04/20/2005 12:00:00 AM</td>\n",
       "      <td>04:00</td>\n",
       "      <td>MISSION</td>\n",
       "      <td>NONE</td>\n",
       "      <td>18TH ST / CASTRO ST</td>\n",
       "      <td>-122.435003</td>\n",
       "      <td>37.760888</td>\n",
       "      <td>(37.7608878061245, -122.435002864271)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>80049078</td>\n",
       "      <td>LARCENY/THEFT</td>\n",
       "      <td>GRAND THEFT FROM A BUILDING</td>\n",
       "      <td>Sunday</td>\n",
       "      <td>01/13/2008 12:00:00 AM</td>\n",
       "      <td>18:00</td>\n",
       "      <td>PARK</td>\n",
       "      <td>NONE</td>\n",
       "      <td>1100 Block of CLAYTON ST</td>\n",
       "      <td>-122.446838</td>\n",
       "      <td>37.762255</td>\n",
       "      <td>(37.7622550270122, -122.446837820235)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>130366639</td>\n",
       "      <td>ASSAULT</td>\n",
       "      <td>AGGRAVATED ASSAULT WITH A KNIFE</td>\n",
       "      <td>Sunday</td>\n",
       "      <td>05/05/2013 12:00:00 AM</td>\n",
       "      <td>04:10</td>\n",
       "      <td>INGLESIDE</td>\n",
       "      <td>ARREST, BOOKED</td>\n",
       "      <td>0 Block of SGTJOHNVYOUNG LN</td>\n",
       "      <td>-122.444707</td>\n",
       "      <td>37.724931</td>\n",
       "      <td>(37.7249307267936, -122.444707063455)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>30810835</td>\n",
       "      <td>DRIVING UNDER THE INFLUENCE</td>\n",
       "      <td>DRIVING WHILE UNDER THE INFLUENCE OF ALCOHOL</td>\n",
       "      <td>Tuesday</td>\n",
       "      <td>07/08/2003 12:00:00 AM</td>\n",
       "      <td>01:00</td>\n",
       "      <td>SOUTHERN</td>\n",
       "      <td>ARREST, BOOKED</td>\n",
       "      <td>MASON ST / TURK ST</td>\n",
       "      <td>-122.408954</td>\n",
       "      <td>37.783288</td>\n",
       "      <td>(37.7832878735491, -122.408953598286)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>130839567</td>\n",
       "      <td>OTHER OFFENSES</td>\n",
       "      <td>TRAFFIC VIOLATION ARREST</td>\n",
       "      <td>Friday</td>\n",
       "      <td>10/04/2013 12:00:00 AM</td>\n",
       "      <td>20:53</td>\n",
       "      <td>TENDERLOIN</td>\n",
       "      <td>ARREST, BOOKED</td>\n",
       "      <td>TURK ST / LEAVENWORTH ST</td>\n",
       "      <td>-122.414056</td>\n",
       "      <td>37.782793</td>\n",
       "      <td>(37.7827931071006, -122.414056291891)</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   IncidntNum                     Category  \\\n",
       "0    50436712                      ASSAULT   \n",
       "1    80049078                LARCENY/THEFT   \n",
       "2   130366639                      ASSAULT   \n",
       "3    30810835  DRIVING UNDER THE INFLUENCE   \n",
       "4   130839567               OTHER OFFENSES   \n",
       "\n",
       "                                       Descript  DayOfWeek  \\\n",
       "0                                       BATTERY  Wednesday   \n",
       "1                   GRAND THEFT FROM A BUILDING     Sunday   \n",
       "2               AGGRAVATED ASSAULT WITH A KNIFE     Sunday   \n",
       "3  DRIVING WHILE UNDER THE INFLUENCE OF ALCOHOL    Tuesday   \n",
       "4                      TRAFFIC VIOLATION ARREST     Friday   \n",
       "\n",
       "                     Date   Time  PdDistrict      Resolution  \\\n",
       "0  04/20/2005 12:00:00 AM  04:00     MISSION            NONE   \n",
       "1  01/13/2008 12:00:00 AM  18:00        PARK            NONE   \n",
       "2  05/05/2013 12:00:00 AM  04:10   INGLESIDE  ARREST, BOOKED   \n",
       "3  07/08/2003 12:00:00 AM  01:00    SOUTHERN  ARREST, BOOKED   \n",
       "4  10/04/2013 12:00:00 AM  20:53  TENDERLOIN  ARREST, BOOKED   \n",
       "\n",
       "                       Address           X          Y  \\\n",
       "0          18TH ST / CASTRO ST -122.435003  37.760888   \n",
       "1     1100 Block of CLAYTON ST -122.446838  37.762255   \n",
       "2  0 Block of SGTJOHNVYOUNG LN -122.444707  37.724931   \n",
       "3           MASON ST / TURK ST -122.408954  37.783288   \n",
       "4     TURK ST / LEAVENWORTH ST -122.414056  37.782793   \n",
       "\n",
       "                                Location  \n",
       "0  (37.7608878061245, -122.435002864271)  \n",
       "1  (37.7622550270122, -122.446837820235)  \n",
       "2  (37.7249307267936, -122.444707063455)  \n",
       "3  (37.7832878735491, -122.408953598286)  \n",
       "4  (37.7827931071006, -122.414056291891)  "
      ]
     },
     "execution_count": 45,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sfpd.limit(5).toPandas() # Using pandas dataframe only for better tabular  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1"
      ]
     },
     "execution_count": 46,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sfpd.rdd.getNumPartitions()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "pyspark.sql.types.Row"
      ]
     },
     "execution_count": 47,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "type(sfpd.first())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['FRAUD', 'SUICIDE', 'LIQUOR LAWS', 'SECONDARY CODES', 'FAMILY OFFENSES', 'MISSING PERSON', 'OTHER OFFENSES', 'DRIVING UNDER THE INFLUENCE', 'WARRANTS', 'ARSON', 'FORGERY/COUNTERFEITING', 'GAMBLING', 'BRIBERY', 'ASSAULT', 'DRUNKENNESS', 'EXTORTION', 'TREA', 'WEAPON LAWS', 'LOITERING', 'SUSPICIOUS OCC', 'ROBBERY', 'SEX OFFENSES, FORCIBLE', 'PROSTITUTION', 'EMBEZZLEMENT', 'BAD CHECKS', 'DISORDERLY CONDUCT', 'RUNAWAY', 'RECOVERED VEHICLE', 'VANDALISM', 'DRUG/NARCOTIC', 'PORNOGRAPHY/OBSCENE MAT', 'TRESPASS', 'NON-CRIMINAL', 'VEHICLE THEFT', 'STOLEN PROPERTY', 'LARCENY/THEFT', 'KIDNAPPING', 'BURGLARY', 'SEX OFFENSES, NON FORCIBLE']\n"
     ]
    }
   ],
   "source": [
    "categories = sfpd.select(\"Category\").distinct().rdd.map(lambda r: r.Category)\n",
    "print(categories.collect())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "39"
      ]
     },
     "execution_count": 49,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "categories_count = categories.count()\n",
    "categories_count"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+---------------------------+------+\n",
      "|Category                   |count |\n",
      "+---------------------------+------+\n",
      "|LARCENY/THEFT              |385774|\n",
      "|OTHER OFFENSES             |269250|\n",
      "|NON-CRIMINAL               |200942|\n",
      "|ASSAULT                    |165324|\n",
      "|VEHICLE THEFT              |114258|\n",
      "|DRUG/NARCOTIC              |111436|\n",
      "|VANDALISM                  |96350 |\n",
      "|WARRANTS                   |89782 |\n",
      "|BURGLARY                   |78968 |\n",
      "|SUSPICIOUS OCC             |67788 |\n",
      "|MISSING PERSON             |55584 |\n",
      "|ROBBERY                    |48713 |\n",
      "|FRAUD                      |36004 |\n",
      "|FORGERY/COUNTERFEITING     |21804 |\n",
      "|SECONDARY CODES            |21636 |\n",
      "|WEAPON LAWS                |18533 |\n",
      "|TRESPASS                   |15798 |\n",
      "|PROSTITUTION               |15657 |\n",
      "|STOLEN PROPERTY            |10096 |\n",
      "|SEX OFFENSES, FORCIBLE     |9541  |\n",
      "|DRUNKENNESS                |9032  |\n",
      "|DISORDERLY CONDUCT         |8992  |\n",
      "|RECOVERED VEHICLE          |6346  |\n",
      "|DRIVING UNDER THE INFLUENCE|4953  |\n",
      "|KIDNAPPING                 |4852  |\n",
      "|RUNAWAY                    |3998  |\n",
      "|LIQUOR LAWS                |3860  |\n",
      "|ARSON                      |3283  |\n",
      "|EMBEZZLEMENT               |2587  |\n",
      "|LOITERING                  |2357  |\n",
      "|SUICIDE                    |1131  |\n",
      "|FAMILY OFFENSES            |1082  |\n",
      "|BAD CHECKS                 |862   |\n",
      "|BRIBERY                    |673   |\n",
      "|EXTORTION                  |620   |\n",
      "|SEX OFFENSES, NON FORCIBLE |335   |\n",
      "|GAMBLING                   |309   |\n",
      "|PORNOGRAPHY/OBSCENE MAT    |47    |\n",
      "|TREA                       |10    |\n",
      "+---------------------------+------+\n",
      "\n"
     ]
    }
   ],
   "source": [
    "category_counts = sfpd.groupBy(\"Category\").count().orderBy(col(\"count\").desc())\n",
    "category_counts.show(categories_count, False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.axes._subplots.AxesSubplot at 0x7f3c64d57cc0>"
      ]
     },
     "execution_count": 51,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAwAAAANSCAYAAADSx6lyAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvNQv5yAAAIABJREFUeJzs3Xm0XXWZ5//3hwAJEYgyWEawDEIUZQp4Gy0FBHEARZEWhYgKrVbsKhWlGxSVXxWtopY4AA5U4YRUo2ChoECVM5SglJhASJgEAgETsRrQiiIBMTy/P86+sHO4Y0xyQ/b7tdZZ9+zv+OwdWOv7nP3d56SqkCRJktQNG0x0AJIkSZLWHhMASZIkqUNMACRJkqQOMQGQJEmSOsQEQJIkSeoQEwBJkiSpQ0wAJEmSpA4xAZAkSZI6xARAkiRJ6pANJzoAaaJstdVWNWPGjIkOQ5IkaVTz5s27u6q2Xh1jmQCos2bMmMHcuXMnOgxJkqRRJbl9dY3lFiBJkiSpQ0wAJEmSpA4xAZAkSZI6xGcAJEmStE558MEHWbJkCffff/9Eh7LWTZkyhW233ZaNNtpojc1hAiBJkqR1ypIlS9hss82YMWMGSSY6nLWmqrjnnntYsmQJ22233Rqbxy1AmhBJtkwyv3n9OsnS1nE1f69NcmGSxzd9ZiRZ3mo3P8kbW2Pu3vR96cSdmSRJ+nPdf//9bLnllp1a/AMkYcstt1zjdz68A6AJUVX3ALMAkpwI3FtVH2+O762qwbqvAG8DTmq6LhqsG8Js4PLm73fXXPSSJGlN69rif9DaOG8TAK3rrgB2Ha1Rev+3HAq8GLgsyZSq6t7GQUmSpFGYAGidlWQSsD/wxVbx9knmt47fUVWXAc8HbquqRUkuBV4GfHOtBStJktaYGcdfvFrHW/zRl6/W8VbFKaecwpw5c5g6depan9tnALQu2qRZ5N8DbAF8v1W3qKpmtV6XNeWzgXOa9+c0x4+SZE6SuUnm3nXXXWsqfkmSpBGdcsop3HfffRMytwmA1kXLm33+TwU2pvcMwLCaOwWvBv4uyWLg08CBSTbrb1tVZ1TVQFUNbL311qs/ckmStN4466yz2HXXXdltt914wxvewO23387+++/Prrvuyv77788dd9wBwFFHHcV55533cL9NN90UgEsvvZR9992XQw89lB133JEjjjiCquK0007jV7/6Ffvttx/77bffWj8vEwCts6pqGXA0cGySkb4M90XANVX1lKqaUVVPBb4BvGptxClJktY/1113HSeddBI/+tGPuOaaazj11FN5+9vfzhvf+EYWLFjAEUccwdFHHz3qOFdffTWnnHIK119/Pbfeeis/+clPOProo3nyk5/MJZdcwiWXXLIWzmZlJgBap1XV1cA1wOFN0fZ9XwN6NL3tPuf3df0G8Lq1GKokSVqP/OhHP+LQQw9lq622AmCLLbbgiiuu4HWv6y0v3vCGN3D55ZePOs6ee+7JtttuywYbbMCsWbNYvHjxmgx7THwIWBOuqk7sO9607/gVrcNNxjjmt4Fv/9nBSZKkTqqqUb+Sc7B+ww035KGHHnq43x//+MeH20yePPnh95MmTeJPf/rTGoh2fLwDIEmSJPXZf//9+frXv84999wDwG9+8xue97zncc45ve8cOfvss9lrr70AmDFjBvPmzQPgW9/6Fg8++OCo42+22Wb8/ve/X0PRj8w7AJIkSVqnTcTXdu600068//3v5wUveAGTJk1i991357TTTuNNb3oTJ598MltvvTVf/vKXAfjrv/5rDj74YPbcc0/2339/Hve4x406/pw5czjwwAOZPn36Wn8OIFW1VieU1hUDAwM1d+7ciQ5DkiT1ueGGG3jmM5850WFMmKHOP8m8qhpYHeO7BUiSJEnqEBMASZIkqUNMACRJkrTO6eo29bVx3iYAkiRJWqdMmTKFe+65p3NJQFVxzz33MGXKlDU6j98CJEmSpHXKtttuy5IlS7jrrrsmOpS1bsqUKWy77bZrdA4TAEmSJK1TNtpoI7bbbruJDmO95RYgSZIkqUNMACRJkqQOMQFQZy1cumyiQ5AkSVrr1okEIMmKJPOTXJvkX5JMbcq3TfKtJDcnWZTk1CQbN3X7JlmW5OokNyb5eGu8o5I8lGTXVtm1SWY076clOasZc1Hzflqr7cwkFzV185JckmSf1th3NfHemOSYIc7nmiRf6ys7M8ltTb+rkvxVq/zQvrb3JpnSjL9Lq/zdSf6xdfydJNsk2TjJKU28NzfXbNtWu8Hre00z9/Oa8g2SnNZcm4VJfp5ku6Zu0yT/1Ix5XZIfJ3lO33iDr+Ob8kuTzG3NO5Dk0r5/r3a/Fw1x7RYnuayvbH6Sa/vKTk2yNMkGzfH/aI37x+Z85if5aP8ckiRJXbZOJADA8qqaVVU7A38E/meSAN8ELqiqmcDTgU2Bk1r9Lquq3YHdgYOSPL9VtwR4/zDzfRG4taq2r6rtgduALwAkmQJcDJzR1D8beAfwtFb/c6tqFvB84P1JnjJYkeSZ9K7rPkke1zfvcU2/44F/GumCVNX9wLuAz6VnG+CtwHubeTYBtqiqpcCHgc2ApzfX6gLgm801hEeu725N/4805YcBTwZ2rapdgEOA/2rqvgD8BphZVTsBRwFb9Y03+Govsp+Y5MBhTuuyvn4/GKbdZoPXtLmeK2kW/YcAvwT2aa7XlwfHBX4F7NccHz/MHJIkSZ20riQAbZcBOwAvBO6vqi8DVNUK4BjgTYN3CAZV1XJgPrBNq/giYKckz2i3TbID8Gzgg63iDwADSbYHjgCuqKpvt8a/tqrO7A+0qu4BbgGmt4pfB/wz8D3glcOc44+bcxxRVX0HuBN4I/Ap4MSq+m1TvS9waXMt/gdwTHONaK7ZA/SuYb/NgcExpgN3VtVDTb8lVfXb5jo8BzihVXdrVV08WszAycAJY2g3kq/TS04AZgNf66vfD7gWOL2plyRJ0hitUwlAkg2BA4GFwE7AvHZ9Vf0OuIO+xXOSJwAz6S2sBz0EfAx4X980zwLmDy6Wm3FX0EsgdmpeV40x3r8EpgALWsWHAefSW7QOtzh9Bb1zHHRye2tMX9t30bvrsXVV/XOr/EDgO/SuxR3NtWmb25wLwCaDW5bofbI/mPx8HXhFU/eJJLs35TvRd436bJKVt/Ic1qq7AnggyX5D9Nu7r9/2w4x/HvDfm/evAC7sqx9MCs6nd+dno2HGkSRJUp91JQHYpFn4zqW3wP8iEGCon39rl++dZAHwa+Ciqvp1X9uvAs8d3Nc+RP/hxn2kMDm/2SP/zVbxYUmuA24FTm2265DkvwF3VdXtwA+BPZrkZNDJzXnOAd7cKj+uvTWmPX9V/Qr4Eb1Pu9ueD1w+xvMZ3LKzI3AAcFaSVNUS4Bn0tgU9BPwwyf5DjNWvfwvQuX31H2LouwD9W4AWDTP+b4DfJjkcuAG47+GT6j0D8jJ6W8N+B/wMeMkYYh7sPyfJ3CRzV9znQ8CSJKl71pUEoL2gfEdV/RG4DhhoN0qyOfAUYHDheFlV7QrsAvxNkv7F85+ATwDvaRVfB+w++PBoM+4GwG70FpvXAXu0xjiE3v73LVpjnNvsi98b+ESSJzXls4EdkyxuYtwceHWr3+BC/8VVtdJDraN4qHkNxvs04JfNdboFeGqSzfr67AFc3z9QVV1Bby//1s3xA1X1b1V1HL1nCV7VXIPd2tdoPKrqR/TujDx3Vfo3zgU+y6O3/xwATAMWNtd5L8axDaiqzqiqgaoamDR12ugdJEmS1jPrSgIwlB8CU5O8ESDJJHqL+TOr6r52w6q6id6Dre951ChwJvAiHlnw3gJczcqfUJ8AXNXUfRV4fpL2/v2VnjlozXsFvf3+72wWy6+h90DtjKqaARzMmtmjPrj9h6r6A/AV4JPNNaK5ZlPp3TlYSZIdgUnAPUn2SPLkpnwDYFfg9uaT+bnA/xl8kDi9b0Y6eBwxngS8exXPD3rbez4GfLevfDbwltY13g54Sf9zIZIkSRraOpsAVFXR+6aX1yS5GbgJuJ9H7+kf9I/0vnlnpd+Nbj4lPw14Yqv4zcDTk9ySZBG9bxh6c9N+OXAQvW8iujXJFfQShA8NM+8/0HsI9+XA0uZbeQb9GHhWkulD9lx1B9AkAI330rs2NzXX6jXAIc01hNaefXqfrB/Z7O9/InBh8xWbC4A/AZ9p+rwFeBJwS5KFwOfpfbvOSuMN91WbVfWvwF19xf3PABza36/V//dV9Q/Nvx8AzSL/pfS+pWmw3R/obYV6xXBjSZIk6RF5ZI2ox4Ikk4GfVNXAqI01osnTZ9YDd9480WFIkiSNKsm81bX+23B1DKK1p6oeoO/ZCK2aXbbxGQBJktQ96+wWIEmSJEmrnwmAJEmS1CEmAJIkSVKHmABIkiRJHWICIEmSJHWICYAkSZLUISYAkiRJUoeYAEiSJEkdYgIgSZIkdYgJgDpr4dJlEx2CJEnSWmcCIEmSJHWICYBWkuQvknw1ya1J5iW5IskhrfpTkyxNskGr7KgklWT/VtkhTdmhzfGlSX6RZH6SG5LMabVdnGSrvjiOSvKZ5v2JSe5L8sRW/b1jjVmSJEmPMAHQw5IEuAD4cVU9raqeDRwObNvUbwAcAvwS2Kev+0Jgduv4cOCavjZHVNUs4PnAPyTZeBzh3Q387/HGLEmSpJWZAKjthcAfq+ofBwuq6vaq+nRzuB9wLXA6Ky/2AS4D9kyyUZJNgR2A+cPMsynwB2DFOGL7EnBYki3GGbMkSZJaTADUthNw1Qj1s4GvAecDByXZqFVXwA+AlwIHA98eov/ZSRYAvwA+WFXjSQDupZcEvHOcMUuSJKnFBEDDSvLZJNck+XmzXedlwAVV9TvgZ8BL+rqcQ2/7zeH0EoV+R1TVrsBfAscmeeo4QzoNODLJ5mOJeZj6OUnmJpm74j6/BUiSJHWPCYDargP2GDyoqrcB+wNbAwcA04CFSRYDe9G3DaiqrgR2BraqqpuGm6Sq7qL3qf1zxhNcVf0X8FXgb8cY81BjnFFVA1U1MGnqtPFML0mStF4wAVDbj4ApSf6mVTa1+TsbeEtVzaiqGcB2wEuSTO0b473A+0aapOmzO7BoFWL8JPBWYMMxxCxJkqQ+JgB6WFUV8CrgBUluS3Il8BXg7+nt7b+41fYPwOXAK/rG+LequmSYKc5OMh+YB5xZVfNadQuSLGlenxwhxrvpPYMweZSY3zOec5ckSeqK9NZPUvdMnj6zHrjz5okOQ5IkaVRJ5lXVwOoYyzsA6qxdtvEZAEmS1D0mAJIkSVKHmABIkiRJHWICIEmSJHWICYAkSZLUISYAkiRJUoeYAEiSJEkdYgIgSZIkdYgJgCRJktQhJgCSJElSh5gAqLMWLl020SFIkiStdSYAkiRJUodMWAKQ5P1JrkuyIMn8JM9pyi9N8oumbH6S85ry05L8f339PzvM2HOS3Ni8rkyyV6uuf/xDm/IVrbL5SWYk2TfJsr7yFzXtK8knWuMem+TE5v0zmnnmJ7khyRlN+UjjDXk9Rrh+Jya5L8kTW2X3tt5vm+RbSW5OsijJqUk2bsVRSV7Ran9Rkn2HmOfMJLe14j26KZ+W5Kxm7EXN+2lN3Ywky5v21zd1G7XG3DPJj5t/hxuTfCHJ1CRHJflM6/yWNmPcmOT0JBu0Yjq0L872nIOvN450DSVJkrpow4mYNMlfAQcBe1TVA0m2AjZuNTmiqub2dTsBmJ/kbKCAtwC7DzH2QcBbgb2q6u4kewAXJNmzqn49wvjLq2pW31gzgMuq6qAhTuMB4L8n+UhV3d1Xdxrwqar6VjPOLq26R403husxnLuB/w28p2+8AN8ETq+qg5NMAs4ATgKOa5otAd4PXDiGeY6rqvP6yr4IXFtVb2zm/D/AF4DXNPWLqmpWM/f3gdcCZyf5C+BfgMOr6oom1lcDmw0x76eq6uPNwv/HwAuAS0aIc1H/v6EkSZJWNlF3AKYDd1fVAwBVdXdV/WqkDlX1O3oL1s8AnwX+rqr+a4im76G3YL276XcV8BXgbasxfoA/0VtUHzNE3XR6C2yaGBaOMta4r0fjS8BhSbboK38hcH9VfbkZb0UT55uSTG3aXAMsS/LiMcyzkiQ7AM8GPtgq/gAwkGT7dttm7iuBbZqitwFfqaormvqqqvOq6j9HmHJjYArw2/HGKkmSpJVNVALwPeApSW5K8rkkL+irP7u1jePkwcKq+hrwBGDzqvrnYcbeCZjXVza3KR9q/C2bsk1aZee32u7dt62kvcD9LHDE4NaXlk8BP0ryb0mOSfL4UcYb7XoM5156ScA7R7sGTQJ1B7BDq/hD9O6sjObkVry7AM8C5jeL+8HxVwDzWfk6k2QK8BzgO03Rzv2xjeCYJPOBO4Gbqmr+KO2377u2e49xHkmSpM6YkC1AVXVvkmcDewP7AecmOb6qzmyaDLVFhyTbAk8CKsmmVXVvf5thhN62oUFj2gLUGG4LEFX1uyRnAUcDy1vlX07yXeAA4GDgrUl2G2m8Ua7HSE6jtzXqE62y/vMdsryqLkvCGBbKK20BSvK0MYy/fbN4nwmcV1ULRj+VRxncArQRcF6Sw6vqnBHaj7oFKMkcYA7ApM23XoWQJEmSHtsm7CHgqlpRVZdW1d8Db6e3D3w0pwInAl8H/n6YNtfT257StkdTviacArwZeFy7sKp+VVVfqqqD6W0X2nmkQVbxetBsg/oq8Let4uuAgXa7JJsDTwEW9Q1xEr2tVeNxHbD74EO5zfgbALsBNzRFg4vxHYDnJnllq2//v8+IqupBencQ9hlnnEONdUZVDVTVwKSp/TduJEmS1n8TkgCk9y05M1tFs4DbR+lzIPBE4Cx6e88PSfKsIZp+DPiHwa09SWYBRwGfWw2hP0pV/YZeQvLmVqwHDH7rTZInAVsCS4cbY6TrkeQjSQ4ZJYxP0nvwefCOzg+BqYPfgtM8iPsJ4Myquq8v/u/R21a1G2NUVbcAV7Py9qETgKuaunbbO4Hjgfc2RZ8BjkzrW46SvL65TkNqHhR+Ho9OXiRJkjROE3UHYFPgK81XRC6gt6f8xFZ9e4/+D5p95KcAf9s8NPoH4N30FpMrqapv09sX/9MkNwKfB17fLERXRf+e/UOHaPMJYKvW8UuAa5NcA3yX3haawW8gGmq8ka7HLsCvGUHzwPP5wOTmuIBDgNckuRm4CbgfeN8wQ5wEbDvSHEN4M/D0JLckWQQ8nVYS1OcCegnJ3s3DvocDH0/va0BvoLf16XdD9Bt8BuBaeslNO4n7pyRLmtcVTVn/MwBHj/OcJEmS1nvprRW1rkry3ap66UTHsT6aPH1mPXDnzRMdhiRJ0qiSzKuqgdFbjs5fAl7Hufhfc3bZxmcAJElS95gASJIkSR1iAiBJkiR1iAmAJEmS1CEmAJIkSVKHmABIkiRJHWICIEmSJHWICYAkSZLUISYAkiRJUoeYAEiSJEkdYgKgzlq4dNlEhyBJkrTWmQBIkiRJHWIC0HFJViSZ33odn2RSknlJ9mm1+16S1yT5WdPujiR3tfrNSDItyVlJFjWvs5JMa/rPSLK8aXt9U7dRkpe2xrg3yS+a92cl2TfJRa0YXpVkQZIbkyxM8qpW3ZlJliaZ3BxvlWTxWryUkiRJjwkbTnQAmnDLq2pWf2GSvwW+kGQP4FCgqupfgH9p6o8CBqrq7a0+5wHXVtUbm+P/A3wBeE3TZFFVzUoyCfg+8NqqOhv4btP+UuDYqprbHO/bGns34OPAi6vqtiTbAd9PcmtVLWiarQDeBJz+518WSZKk9ZN3ADSkqvoZ8FPgRODDwNtGap9kB+DZwAdbxR8ABpJs3zf2CuBKYJtxhHQs8OGquq0Z4zbgI8BxrTanAMckMbGVJEkahgmANunbAnRYq+69wLuAr1bVLaOM8yxgfrO4Bx5e6M8Hdmo3TDIFeA7wnXHEuRMwr69sbt/YdwCXA28YbpAkc5LMTTJ3xX0+BCxJkrrHT0o15Bagxj7AMmDnMYwToEYp3z7JfGAmcF5r685YDDX+UGUfBr4NXDzUIFV1BnAGwOTpM4eKV5Ikab3mHQANKcnjgI8BLwS2TvKyUbpcB+ye5OH/ppr3uwE3NEWLmmRjB+C5SV45jpCuAwb6yvYArm8XNHcq5gOvHcfYkiRJnWECoOH8HfD1qroR+FvgU83WnSE1C++rgRNaxScAV/VvH6qqO4Hj6W0xGquPA+9NMgN63yoEvA/4xBBtT6L3zIAkSZL6mACo/xmAjyZ5FnAIvYU0VTWf3jf1vGeUsd4MPD3JLUkWAU9vyoZyATA1yd5jCbKJ4T3AhUluBC4E3t2U97e9DrhqLONKkiR1TarcBq1umjx9Zj1w580THYYkSdKoksyrqv7t0KvEOwDqrF22mTbRIUiSJK11JgCSJElSh5gASJIkSR1iAiBJkiR1iAmAJEmS1CEmAJIkSVKHmABIkiRJHWICIEmSJHWICYAkSZLUISYAkiRJUoeYAKizFi5dNtEhSJIkrXUmAJIkSVKHmABolSVZkWR+kmuSXJXkeU35jCTLm7rrk5yVZKOmbt8kFzXvj0pyV9PuuiTnJZna1J2YZGlTN/h6fNN/WZKrk9yY5ONN+52S3JRkk1Z8Fyc5fO1fGUmSpHWXCYD+HMuralZV7Qa8F/hIq25RVc0CdgG2BV47zBjnNmPsBPwROKxV96mmbvD1X035ZVW1O7A7cFCS51fVdcA3gfcDJHkVsFFVnbO6TlaSJGl9sOFEB6D1xubAb/sLq2pFkiuBbUbqnGRD4HFDjTGcqlqeZH5r7A8AVyc5D/go8IqxjiVJktQVJgD6c2zSLMCnANOBF/Y3SDIFeA7wzmHGOCzJXk3/m4ALW3XHJHl98/63VbVf39hPAGYCPwaoqvuSHNscf7Kqbh4injnAHIBJm2891vOUJElab7gFSH+OwS1AOwIHAGclSVO3fZMc3APcUVULhhnj3Gar0JOAhcBxrbr2FqD24n/vJAuAXwMXVdWvByuq6kLgv4DPDTVZVZ1RVQNVNTBp6rRVOGVJkqTHNhMArRZVdQWwFTD4sfrgMwA7AM9N8spR+he9T//3GcN0l1XVrvSeL/ibJLP66h9qXpIkSepjAqDVIsmOwCR6n/g/rKruBI6n95DwaPYCFo11zqq6id6Dx+8Ze6SSJEnd5jMA+nMMPgMAEODI5qHf/nYXACcm2XuIMQafAdgAWAIc1aprPwMA8Koh+v8jcGyS7arqtlU5CUmSpC5Jb+eF1D2Tp8+sB+581HPCkiRJ65wk86pqYHWM5RYgddYu2/gQsCRJ6h4TAEmSJKlDTAAkSZKkDjEBkCRJkjrEBECSJEnqEBMASZIkqUNMACRJkqQOMQGQJEmSOsQEQJIkSeoQEwBJkiSpQ0wA1FkLly5jxvEXT3QYkiRJa5UJgCRJktQhJgAdkGRFkvlJrklyVZLn9dUfk+T+JNNaZfsmWZbk6iS/SPLjJAeNMMeBSeYmuSHJjUk+3pSfmOTYvraLk2zVF9vg6/imfKMkH01yc5Jrk1yZ5MAh+j87yW1Jdk/yF0kuas7z+iT/urquoSRJ0vpiw4kOQGvF8qqaBZDkpcBHgBe06mcDPwcOAc5slV9WVQc1/WYBFyRZXlU/bA+eZGfgM8DLq+rGJBsCc8YbW58PAtOBnavqgSR/0RczSXYFzgMOq6qrk/wT8P2qOrVVL0mSpBbvAHTP5sBvBw+SbA9sCpxALxEYUlXNBz4AvH2I6ncDJ1XVjU3bP1XV51Y1wCRTgb8G3lFVDzRj/mdVfb3V7JnABcAbqurKpmw6sKQV84JVjUGSJGl95R2AbtgkyXxgCr1F8gtbdbOBrwGXAc9I8sSq+n/DjHMVcNwQ5TsDnxhh/mOSvL51/OQhYhv0EeAG4I6q+t0IY34LeH1VXd4q+yxwbpK3Az8AvlxVv2p3SjKH5u7EpM23HmF4SZKk9ZN3ALpheVXNqqodgQOAs5KkqTscOKeqHgK+CbxmhHEyQt1IPtXMP6vZ7tNelC9v11XVuWMc8wfAW5JMGiyoqu8CTwM+D+wIXJ1kpVV+VZ1RVQNVNTBp6jQkSZK6xgSgY6rqCmArYOtmj/xM4PtJFtNLBobdBgTsTu/T+X7XAc9ejWHeAvxlks1GaDO4FWmlrUZV9Zuq+mpVvYHecw37rMa4JEmSHvNMADomyY7AJOAeeov9E6tqRvN6MrBNkqcO0W9X4P+jt82m38nA+5I8vWm7QZL/taoxVtV9wBeB05Js3Iw5vW8b0UNN/M9I8oGmzQub5wdokoftgTtWNQ5JkqT1kc8AdEN7n32AI6tqRZLDgQP72p5P707Az4C9k1wNTAX+H3B0/zcAQe9h2yTvAr7WLMALGOsvbPU/A/Cdqjqe3kPJHwKuT3I/8Afg7/rmfSDJwcC/J/nPJs7PJPkTveT2C1X18zHGIUmS1AmpqomOQZoQk6fPrOlHnsLij758okORJEkaUZJ5VTWwOsZyC5A6a5dtprn4lyRJnWMCIEmSJHWICYAkSZLUISYAkiRJUoeYAEiSJEkdYgIgSZIkdYgJgCRJktQhJgCSJElSh5gASJIkSR1iAiBJkiR1iAmAOmvh0mXMOP7iiQ5DkiRprTIBkCRJkjrEBGACJVmRZH7rNaNVd2qSpUk2aJUdlaSS7N8qO6QpO7Q5vjTJQPN+cZKtWm13TXJjkimtsu8O9u2LbZ8kP2/a35jkza26DzWxDcZ9UlN+eZJftMoPSbLhEOd5XKv9z1rjPjfJD5r3myY5J8nCJNcmuSzJ1FHGO7g5vibJ9Une8uf8+0iSJK2PNpzoADpueVXN6i9sFv2HAL8E9gEubVUvBGYDP2yODweuGctkVbUgyUXA8cCJzcL/oao6r2/+JwP/F3hlVc1PsjXwvSRLq+o7TbOTq+qUIaY5rKrmt8baEPj9UOfZmJ7kxVX1/b7yY4A7qurwZpwdgQebukeNl2QycDowUFW/ao6fOuIFkSRJ6iDvAKyb9gOupbegnd1XdxmwZ5KNkmwK7ADMZ+z+Hnhdkl2Bk4C3DdHmHcAXBhfyVXUXvaThPeM6i7E5GThhiPLpwNLBg6q6saoeHKLdoGlAgN807R/noerRAAAgAElEQVSoqptWZ6CSJEnrAxOAibVJaxvL+a3y2cDXgPOBg5Js1Kor4AfAS4GDgW+PZ8Kq+gO9hfzlwD9X1a1DNNsJmNdXNrcpH3RcK/YXtcrPbZU/vinbrG/LTnvL0WUASfbum++LwAlJfprkg0l2aNU9aryq+n/Ad4Hbk3w1yez29qlBSeYkmZtk7or7lg19kSRJktZjbgGaWI/aApRkY+BlwDFV9ftmj/xLgPbX1ZwDHE3vU+//DbxvPJNW1flJPgt8bpgmoZdojFS2urYAQe9OxAn07k4MxjgvydPonfuLgLlJ9gRuHW68qjqqubPxInp3LPYH3tLX5gzgDIDJ02f2n6MkSdJ6zzsA654D6C3sFyZZDOxF3zagqroS2BnY6s/Y5vJQ8xrKdcBAX9kewPWrONeIqup7wOP756yq31fVN6rqb+glPQeOYawFVfVJendIXr0m4pUkSXosMwFY98wG3lJVM6pqBrAd8JIkU/vavZdxfvI/Dp8B3tJ8mk7zTUIfAT62huaD3l2Adw8eJNlrcAtR80DvM4Hbh+ucZPMk+7SKZo3UXpIkqavcArQOaRb5LwXeOlhWVX9Icjnwinbbqvq3MQ67IMngJ/1fr6r/NVqHqlqS5EjgS82DxgCfHMec/TZL0n5Q+eKqen9fmwuBD7SOZwKnJ4Feonoh8C1g0lDj0UtO3pvk88By4F7gTasYryRJ0norVW6DVjcNDAzU3LlzJzoMSZKkUSWZV1X9W7RXiVuAJEmSpA4xAZAkSZI6xARAkiRJ6hATAEmSJKlDTAAkSZKkDjEBkCRJkjrEBECSJEnqEBMASZIkqUNMACRJkqQOMQGQJEmSOsQEQJ21cOkyZhx/8USHIUmStFaZAGi1SPL+JNclWZBkfpLnJFmcZKtWm32TXNS8PyrJZ1p1b0xybTPG9UmObcrPTHJo8/7SJL9o5rgxyWeSPL41xopm7sHX8WvvCkiSJD02bDjRAeixL8lfAQcBe1TVA82if+Nx9D8QeBfwkqr6VZIpwBuGaX5EVc1NsjHwEeBbwAuauuVVNWuVT0SSJKkDvAOg1WE6cHdVPQBQVXdX1a/G0f+9wLGDfarq/qr6/EgdquqPwLuBv0yy2yrGLUmS1DkmAFodvgc8JclNST6X5AWj9ljZzsC88U5aVSuAa4Adm6JN+rYAHTbeMSVJktZ3bgHSn62q7k3ybGBvYD/g3Gb/fQ3VfDVPn9b7UbcAJZkDzAGYtPnWqzkUSZKkdZ93ALRaVNWKqrq0qv4eeDvwauAe4AmtZlsAdw/R/Trg2eOdM8kkYBfghnHEeUZVDVTVwKSp08Y7pSRJ0mOeCYD+bEmekWRmq2gWcDtwKc3DvM1i/fXAJUMM8RHgY0me1LSdnOToUebcqOn3y6pa8GefhCRJUke4BUirw6bAp5uv5PwTcAu9bTYPAqcnuYbeVp3vAP+3v3NV/WuSvwB+kCT0tgl9aZi5zk7yADAZ+AFwcKtukyTzW8ffqSq/ClSSJKklVat7S7b02DB5+syafuQpLP7oyyc6FEmSpBElmVdVA6tjLLcAqbN22Waai39JktQ5JgCSJElSh5gASJIkSR1iAiBJkiR1iAmAJEmS1CEmAJIkSVKHmABIkiRJHWICIEmSJHWICYAkSZLUISYAkiRJUoeYAEiSJEkdsuFEByBNlIVLlzHj+IsfPl780ZdPYDSSJElrh3cAtJIk9w5TPifJjc3ryiR7teouTTKQ5GdJ5ie5I8ldzfv5SWYkWZxkYavstKbvmUlua8quSbJ//7jN+8VJvtGqOzTJma3jA5q4bmzGOjfJX66BSyRJkvSY5h0AjSrJQcBbgb2q6u4kewAXJNmzqn492K6qntO0PwoYqKq3t8YA2K+q7h5iiuOq6rwk+wFnADOHCWUgyU5VdV1ffDsDnwZeWVU3NGWvBGYAd6zKOUuSJK2vvAOgsXgPvUX63QBVdRXwFeBtq3meK4BtRqj/OPC+YeL78ODiH6Cqvl1VP17N8UmSJD3mmQBoLHYC5vWVzW3Kx+OS1hagY4aoPwC4YIT+Xwf2SLLDEPFdNc5YJEmSOsktQFpVAWqcfYbbAnRyko8BTwSeO0L/FcDJwHuBfxsyqGRL4IfAVOCMqvp4X/0cYA7ApM23Hmf4kiRJj33eAdBYXA88u69sj6Z8dTgO2AE4gd7WopH8M7AP0H7A97omHqrqnqqaRe9Zgk37O1fVGVU1UFUDk6ZOWx2xS5IkPaaYAGgsPgb8Q/PpOklmAUcBn1tdE1TVQ8CpwAZJXjpCuweBTwHv6ovv/Ume2SqburpikyRJWp+4BUj9piZZ0jr+ZFV9Msk2wE+TFPB74PVVdec4x74kyYrm/YKqemO7sqoqyYeAdwPfHWGcL9K7WzDYb2GSdwJnJdkMuIfet//8/TjjkyRJWu+larzbuKX1w+TpM2v6kac8fOwPgUmSpHVVknlVNbA6xvIOgDprl22mMddFvyRJ6hifAZAkSZI6xARAkiRJ6hATAEmSJKlDTAAkSZKkDjEBkCRJkjrEBECSJEnqEBMASZIkqUNMACRJkqQOMQGQJEmSOsQEQJIkSeqQDSc6AGmiLFy6jBnHX/zw8eKPvnwCo5EkSVo7vAOwjkuyIsn81uv4pvzSJHckSavtBUnubd7PSLK86XNNkp8meUZTt2+SZX3jvijJIX1l85M8lOTAJG/rK782SSV5ZjPeRUPEfmmSX7T6nNeUn9j03aHV9pimbKA5XpxkYavvaU35mUmWJpncHG/VtN2l1fY3SW5r3v9gzf3rSJIkPfZ4B2Ddt7yqZg1T91/A84HLkzwemN5Xv2iwb5K3Au8DjmzqLquqg4YY8/zBN0nmAEcA362qh4DPtuo+DMyvqhuS/MUI8R9RVXOHKF8IHA58qDk+FLi+r81+VXX3EH1XAG8CTh8sqKqFwOC5nglcVFXnjRCXJElSJ3kH4LHtHHqLaID/DnxzhLabA78d68BJng78HfCGZvHfrtsHeC3wt+OKdmUXAAc34z0NWAbcNca+pwDHJDGBlSRJGicTgHXfJn1bbw5r1f0Q2CfJJHqJwLl9fbdv+iwC/hfwyVbd3n3jbj9YkWQj4KvAsVV1R3vA5k7Dl4Ejq+p3Y4j/7NYcJ7fKfwf8MsnOwOwhYge4pNX3mFb5HcDlwBvGML8kSZJa/AR13TfSFqAV9BbChwGbVNXi1iMBsPIWoMOAM4ADmrrhtgABfBC4rqrOGaLudOD/VtVPxhj/cFuA4JE7GC8F9gf+R1/9cFuAAD4MfBu4eJj6ITXbmuYATNp86/F0lSRJWi94B+Cx7xzg08DXR2n3bWCf0QZLsi/wauDtQ9QdCcyglyCsDhfS+xT/jjHeTXhYVd0CzKe3FWk8/c6oqoGqGpg0ddp4ukqSJK0XvAPw2HcZ8BHga6O02wtYNFKDJE+gt73ndVX1+766pwEnAftU1Z9WPdxHVNXyJO8BblrFIU5inHcAJEmSus4EYN23SZL5rePvVNXxgwdVVcDHh+m7fdM3wB+Bt7Tq9u4b90PATOCJwOl9W4k+ArwQeBzwzb66dzR/90+ypFX+mubv2UmWN+/vrqoXtTsPs81o0CVJVjTvF1TVG/v6XpfkKmCPEcaQJElSS3rrR6l7Jk+fWdOPPOXhY38ITJIkrauSzKuqgdUxlncA1Fm7bDONuS76JUlSx/gQsCRJktQhJgCSJElSh5gASJIkSR1iAiBJkiR1iAmAJEmS1CEmAJIkSVKHmABIkiRJHWICIEmSJHWICYAkSZLUISYAkiRJUodsONEBSBNl4dJlzDj+4pXKFn/05RMUjSRJ0trhHQCtMUkOSVJJdmyOZyRZnmR+kuuTnJVko6ZuapKzkyxMcm2Sy5Ns2tRtm+RbSW5OsijJqUk2bur2beZ4RWvei5LsOwGnLEmStM4zAdCaNBu4HDi8VbaoqmYBuwDbAq9tyt8J/GdV7VJVOwNvBh5MEuCbwAVVNRN4OrApcFJrzCXA+9fomUiSJK0nTAC0RjSf3j+f3kL+8P76qloBXAls0xRNB5a26n9RVQ8ALwTur6ovt/odA7wpydSm+TXAsiQvXkOnI0mStN4wAdCa8irgO1V1E/CbJHu0K5NMAZ4DfKcp+hLwniRXJPlQkplN+U7AvHbfqvodcAewQ6v4Q8AJq/80JEmS1i8mAFpTZgPnNO/PaY4Btk8yH7gHuKOqFgBU1XzgacDJwBbAz5M8EwhQQ4y/UnlVXQaQZO+RgkoyJ8ncJHNX3LdsVc9NkiTpMctvAdJql2RLelt3dk5SwCR6i/XP0TwDkGQ6cGmSV1bVtwGq6l56+/2/meQh4GX0tve8um/8zYGnAIuALVtVJ9F7FuBPw8VWVWcAZwBMnj5zqMRCkiRpveYdAK0JhwJnVdVTq2pGVT0FuI3eQ78AVNWdwPHAewGSPD/JE5r3GwPPAm4HfghMTfLGpm4S8AngzKq6rz1pVX0PeAKw2xo+P0mSpMcsEwCtCbOB8/vKvgG8r6/sAnqL+72B7YF/T7IQuBqYC3yjqgo4BHhNkpuBm4D7hxhr0Em0Eg1JkiStLL31ldQ9k6fPrOlHnrJSmT8EJkmS1kVJ5lXVwOoYy2cA1Fm7bDONuS74JUlSx7gFSJIkSeoQEwBJkiSpQ0wAJEmSpA4xAZAkSZI6xARAkiRJ6hATAEmSJKlDTAAkSZKkDjEBkCRJkjrEBECSJEnqEBMASZIkqUM2nOgApImycOkyZhx/8aPKF3/05RMQjSRJ0trhHYD1TJJ7hyg7McmxzfskOSHJzUluSvLvSXYdrn+So5J8pnU8J8mNzevKJHu16i5N8osk1yT5eZJZw8R4aZKBYepOTbI0yQateO9O8oTmeHqS6pv3riRbJnlGM/b8JDckOWOs102SJKkrTAC6523A84DdqurpwEnAhUkeN1rHJAcBbwX2qqodgf8JfDXJk1rNjqiq3YDPASePJ7Bm0X8I8EtgH4CqKuBnwF81zZ4HXN38JckzgLur6h7gNOBTVTWrqp4JfHo880uSJHWBCUD3vAd4R1XdB1BV3wN+DBwxxr7HVdXdTd+rgK/QSyr6XQFsM87Y9gOuBU4HZrfKf0Kz4G/+fpKVE4KfNu+nA0sGO1XVwnHOL0mStN4zAeiQJJsDj6uqRX1Vc4FnjWGInYB5Q/TdaYi2BwAXjDPE2cDXgPOBg5Js1JT/lEcSgD2bcZ/SHD+PXoIA8CngR0n+LckxSR4/zvklSZLWeyYAAsgo9TVK33b92UmW0LtbMOYtOEk2Bl4GXFBVv6O37eclTfWVwO7NNqWNqupe4NYkO9C6A1BVXwaeCfwLsC/wH0km980zJ8ncJHNX3LdsrOFJkiStN0wAOqRZWP8hydP6qvag90k+wPJmMT5oC+Du5v31wLOH6Ht96/gIYDvgq8BnxxHeAcA0YGGSxcBeNNuAmu1KtwBvAq5q2v8HvYThicAvWuf4q6r6UlUdDPwJ2Lk9SVWdUVUDVTUwaeq0cYQnSZK0fjAB6J6TgdOSbAKQ5EX0tvCc19T/O/D6pm4T4LXAJU3dx4B/SLJlUz8LOIreA78Pq6oHgROA5yZ55hjjmg28papmVNUMeknES5JMbep/AryL3rMFNH/fCfxH86AwSQ4Y3DbUPJi8JbB0jPNLkiR1gr8DsP6Z2mzBGfTJvvpPA48HFjSL5Y2Bnavq/qb+ncA/JTma3vaes6rqxwBV9e0k2wA/TVLA74HXV9Wd/UFU1fIknwCOBd48RJwXJ3mweX8FsD+9bxga7P+HJJcDrwDOpZcAvJNHEoCrgG2BL7TGfAlwapLBczmuqn49xNySJEmdlebDU3VQkk3pPXD786p630THs7ZNnj6zph95yqPK/SEwSZK0rkkyr6qG/B2l8fIOQIc1D9O+eKLjmCi7bDONuS72JUlSx/gMgCRJktQhJgCSJElSh5gASJIkSR1iAiBJkiR1iAmAJEmS1CEmAJIkSVKHmABIkiRJHWICIEmSJHWICYAkSZLUIf4SsDpr4dJlzDj+4keVL/bXgSVJ0nrMOwCSJElSh5gAaEhJViSZn+TaJBcmeXxTvm+Si/ranpnk0Ob9pUnmtuoGklza1/7UJEuTbNAcJ8ndSZ7QHE9PUkn2avW5K8mWreNrknytdTwnybmt482TLEqy3Wq6JJIkSesFEwANZ3lVzaqqnYHfAG8bR98nJjlwqIpm0X8I8EtgH4CqKuBnwF81zZ4HXN38JckzgLur6p7m+Jn0/tvdJ8njmj6fB7ZN8qLm+APAl6rqtnHELUmStN4zAdBYXAFsM472JwMnDFO3H3AtcDowu1X+E5oFf/P3k6ycEPy01fZ1wD8D3wNeCQ8nEX8DnJJkANi/iUOSJEktJgAaUZJJ9BbT3x5HtyuAB5LsN0TdbOBrwPnAQUk2asp/yiMJwJ7ABcBTmuPn0UsQBh0GnNuM83ASUVULgO8CPwSOrqo/jiNmSZKkTjAB0HA2STIfuAfYAvh+U17DtO8v/xB9dwGSbAy8DLigqn5Hb9vPS5rqK4Hdmy09G1XVvcCtSXagdQcgyX8D7qqq2+kt9PcYfHag8VlgaVVdMlSQzbMCc5PMXXHfspGvgCRJ0nrIBEDDWV5Vs4CnAhvzyDMA9wBP6Gu7BXB3u6CqfgRMAZ7bKj4AmAYsTLIY2IvmE/yqug+4BXgTcFXT/j/oJQxPBH7RlM0Gdmz6LwI2B17dmuOh5jWkqjqjqgaqamDS1GnDn70kSdJ6ygRAI6qqZcDRwLHNdp2bgSc3D+KS5KnAbsD8IbqfBLy7dTwbeEtVzaiqGcB2wEuSTG3qfwK8i94WIpq/7wT+o6qqeYD4NcCurTEOZuVnCSRJkjQCEwCNqqquBq4BDq+qB4DXA19utgidR29R/6j9NFX1r8BdAM0i/6XAxa36PwCXA69oin4CPI1HEoCrgG155AHgfeht71namubHwLOSTF8NpypJkrTeS+/LU6TumTx9Zk0/8pRHlftLwJIkaV2TZF5VDayOsTZcHYNIj0W7bDONuS72JUlSx7gFSJIkSeoQEwBJkiSpQ0wAJEmSpA4xAZAkSZI6xARAkiRJ6hATAEmSJKlDTAAkSZKkDjEBkCRJkjrEBECSJEnqEH8JWJ21cOkyZhx/8ZB1i/2FYEmStJ7yDoAkSZLUISYAHZPk3tb7lyW5OclfJjkxybFN+ZlJbktyTZKbkpyVZJtWv8VJvtE6PjTJmX3zfCvJFX1lJyZZmmR+kmuTvHIM5e2YliaZ3BxvlWRxa+yZSS5KsijJvCSXJNln9V05SZKk9YMJQEcl2R/4NHBAVd0xRJPjqmo34BnA1cAlSTZu1Q8k2WmYsR8P7AE8Psl2fdWfqqpZwGuALyXZYJTythXAm4aYbwpwMXBGVW1fVc8G3gE8bciTlyRJ6jATgA5KsjfweeDlVbVopLbV8yng18CBraqPA+8bpturgQuBc4DDhxn3BuBPwFZjKW+cAhyTpP/ZlSPg/2fv7uMum+v9j7/eBmMmjJuoaVQTJoXJYH4p0aEQpSPlJCmm1JxOOWocakInichNhujUpJIOUhIiUQc/1FSuYW6MuyGTY0q5+TXuBpl5//5Y382aPXtf175mrms0s9/Px2M/Zq/v3fqsdc0f67PW97s202xfXhvnNtvntokvIiIiomslAeg+Q4HLgPfYvrMf/W4BXlfb/hGwnaTNW7Q9ALiwfA5oNZikHYDFwEOdlBf3AzcBH24q36rEFxERERF9SALQff4O/AY4pJ/91LS9CDgF+PwSjaSXAZsDN9m+G3hO0ta1JpMkzaB6grC/bfdR3uwrwJH08n9X0k/LWoJLWtRNlNQjqWfRUwvaHmxERETEqioJQPdZDLwf+D+S2k3haWVb4I6msh8AbwVeVSvbH1gfuK8s0h3NktOATrc9zvbOtm/soHwJtu8BZpRjaJhDteag0WZfYAKwQYv+U22Ptz1+yPAR7XYTERERscpKAtCFbD8F7A0cKKnXJwGqHAaMBH7RNM7fgdOBz9SKD6BaWDza9mhge9qsA1gOJwBH1LYvAN7SeHtQMXyA9xkRERGxSkgC0KVsPwrsCRwjaZ8WTU6RNBO4G/g/wK62n23R7juUH5STNJrqacBva/u5D3iszO0fqNjnUJvzb3shVULzCUl/KK8fPQY4fqD2GREREbGqUPup1hGrtqEjx3jkwVNa1uWXgCMiIuIfiaTptscPxFjNr1OM6BpjR42gJxf6ERER0WUyBSgiIiIiooskAYiIiIiI6CJJACIiIiIiukgSgIiIiIiILpIEICIiIiKiiyQBiIiIiIjoIkkAIiIiIiK6SBKAiIiIiIgukgQgIiIiIqKL5JeAo2vNnr+A0ZOvbFk3L78QHBEREauoPAGIiIiIiOgig5YASFokaYakOZJmSjpc0mqlbhdJCyTdKulOSafW+k2QdFZpM61pzNUl/UXSSEnnStqvlF8vqafWbryk62vbbyxt5kq6RdKVksa2iPlYSUc0lc2T9NLy3ZJOq9UdIenYWt/55ZjnSrpE0pa1ttdLuqvUz5B0cYt+t0s6oEVcR9f6Lap9P6yDmOvtZ0ia3GL8js5l7e/WGOtXvZy30ZJua3d+yz7vq431m1I+QdJiSW+o9btN0ujyfW1J35J0b/m/dYOkHTo91oiIiIhuN5hTgBbaHgcgaWPgAmAE8MVSf6PtvSUNA26V9FPbv671vwHYRNJo2/NK2W7Abbb/LKl5fxtL2sv2VfVCSS8DfgR80HbjInMnYDNgdj+P6RngvZJOtP1wi/rTbZ9a9rE/cK2ksbYfKvUH2u5p10/SGGC6pItt/71RafsE4IQy7hON81q2j+0j5oX19h1qeS6LG23v3c/x2jnS9sUtyh8Ajgb2b1F3DnAfMMb2YkmbAq8vdctyrBERERFdZYVMAbL9V2AicKiartxtLwRmAKOayhcDP2bJi8APABe22c0pwDEtyg8Fvt+4+C9j32T70v4eB/AcMBWY1FdD2xcB1wAf7HRw23OBp4D1lyG2gdTuXK4oVwBbSdqiXihpM2AH4Jjy/wPbf7DdeiJ/RERERCxlha0BsP2Hsr+N6+WS1gfGUN3xb3Yh1UU/koYC7wR+0mYX04BnJO3aVL4VcMuyR76Us4EDJY3ooO0twOtq2+fXpqec0txY0nbA3JIw9cek+tQX4BW1umFN02Ja3VVv1u5cAuxcG+vofsbZ7JTaWOfXyhcDJwNHNbXfCphhe1Gb8fo8VkkTJfVI6ln01ILlDD8iIiJi5bOi3wJUv/u/s6RZwBbASbYfbG5s++Yy53sLqmkev7X9/3oZ/3iqO9efaxuA9DtgXeAa259u3mWbbs+X235M0nnAYcDCXmKBJY8X2k8BmiTp48CmwJ59jNnK81OPoFoDUKtb1mkx7c5lp1OA+jyXtJ8CBNWUsaMlvaaDfTX0eay2p1I9xWHoyDHtYoyIiIhYZa2wJwBlrvYioHF3+0bbbwDGAv8mqd2F2w+pngL0Nv0HANvXAmsBb6oVzwG2q7XZAfgC1XqEZo+w9PSbdYC/NZVNAQ4BXtJbPMC2wB19tIHqAn4LqulO50laq4M+g6rNueyPVudyA6DV2olW+38OOI0lE5A5wDYqi8kjIiIiov9WyIWUpI2AbwJn2V7irqvtu4ETaX/X/kLgQ8DbgMs72N0JwGdr22cDEyTtWCsb3qbvDcA/S1qnxP1eYGbzlBPbj1ItLD6kXRCS3gfsQR9JS9O4lwA9wMGd9hlkzeeyY7afAP4s6e0AkjagerpxUz+GOZdq4fdGZcx7qc7PlxprSSSNkbTPssQYERER0Y0GcwrQsDIffQ2qxbM/AL7Wpu03gSNaTfewfbukp4Dptp/sa6e2fy7podr2g2Uu+FcljaJ6AvEwcFyLvrMknQXcJMml7cfa7Oo0qgXGdZMkfYjqycBtwNtqbwCCag1AY9rQw7Z3azHuccAFkr7dWOi6nBp/h4Zf2O7o9ZjN57IPx0j6TK3vJsBBwNl64dWpXyoX8Q2nSKovNn5j0/6flXQmcEat+GNU5/6e8v/iEeDIUrfMxxoRERHRLdR0Qz6iawwdOcYjD57Ssi6/BBwRERH/SCRNtz1+IMZa0YuAI/5hjB01gp5c6EdERESXyWLKiIiIiIgukgQgIiIiIqKLJAGIiIiIiOgiSQAiIiIiIrpIEoCIiIiIiC6SBCAiIiIiooskAYiIiIiI6CJJACIiIiIiukgSgIiIiIiILpJfAo6uNXv+AkZPvrLXNvPyS8ERERGxiskTgIiIiIiILpIEYBlJWiRphqTbJP1M0nqlfLSkhaWu8Tmo1K0t6VuS7pU0R9INknYodZtIukzS3FJ/hqQ1Jb1E0iOSRjTt/1JJ75c0QdJDTfvbsimO2yWdJ2mN0ncXSQua+uzWdFxzJM2UdLikpf6fSLpP0hZNZVMkfbaP8Z9o6jNB0lnl+7GSjqjVHSHpznKOZ9bO4/WSxjeN03afEREREfGCJADLbqHtcba3Bh4FPlWru7fUNT7nlfJzStsxtrcCJgAvlSTgEuBS22OA1wJrAyfYfhK4BnhPY/CSDOwEXFGKLmra3+31OICxwCbA+2sx3tjU51dNx7UVsDvwTuCLLY7/h8AHajGtBuwHXNTH+B2R9Imy/zeWc/xWQH10W659RkRERHSDJAADYxowqrcGkjYDdgCOsb0YwPYfbF8JvA142vb3SvkiYBLwUUnDgQupXWwD+wK/sP1UJ8GV8X7fV4wt+v0VmAgcWpKUuuaY3grMs/3H/uyjF0cBn7T9WIllge3vD9DYEREREV0ri4CXk6QhwNuB79SKN5M0o7b978D6wIxyMd5sK2B6vcD2Y5LuBzYHfgGcI2lD249QXXh/vdZ8f0k71bbf3BTjWlTJx6drxTs3xfg+2/c2B2b7D+Xu/sbAX2rlsyQtlrSN7Zklpgs7GH9YU1CA/z4AACAASURBVPkGwOVN8a4DrNMqnj70eUySJlIlNQxZd6N+Dh8RERGx8ksCsOwaF7KjqS7ef1mra0y9eZ6kf+5lLAFuV277WUmXA/tJ+gkwjmpaUMNFtg9t2h+8kIiMAS62PavW5Ebbe/d2gE1xtHIh8AFJc4B9gP/sYPyF9XMjaQIwvqlNu/PRlz6PyfZUYCrA0JFjlmUfERERESu1TAFado0L2VcDa7LkGoBW5gDbtFpQW+qaF7WuC7wSaNzBbky52Q+4zPbfO4ixkYhsDrypjySkJUmbAouAv7aovpBqXcFuwKwyZWi5lWk/T5Z9R0RERMQASgKwnGwvAA4Djmi8ZadNu3uBHuBLjfn0ksZI2gf4H2B47S03Q4DTgHNr8/yvo7qT/ymWnGrTSYx/BiYDn+9PP0kbAd8EzrK91N3yckyPACf1N6YOnAicXRIhJK1bpu9ERERExHJIAjAAbN8KNObBQ5l6U/scVso/BrwcuEfSbODbwJ/KxfW+wL9ImgvcDTxNtRC2sY/FwE+ADYEbmkLYv2l/O7YI81KqJGPnsr1zU5/9SvmwxmtAgV9RTTX6Ui+HfyHwOuCnTeXtxu/Uf1ElPTdLug34v0B90fOVkh4onx8P0D4jIiIiVnlqcWM3oisMHTnGIw+e0mub/BJwRERE/COQNN1287rJZZJFwNG1xo4aQU8u8CMiIqLLZApQREREREQXSQIQEREREdFFkgBERERERHSRJAAREREREV0kCUBERERERBdJAhARERER0UWSAEREREREdJEkABERERERXSQJQEREREREF8kvAUfXmj1/AaMnX9mvPvPyy8ERERGxkssTgIiIiIiILpIEoImkRZJmSJojaaakwyWtVup2kXRF+f4ySVeUNrdL+nltjK0kXSvpbklzJX1BkkrdBEkPlX3cKWlSrd+xkuaXurmSLpG0Za3+ekl3lfoZki5u0e92SQfU+pwrab/a9vASV33coySd1eJcvELSjyTdU8a9UtLmpW5sLZ65ko6q9fuYpMWStqqV3Slpk/L9AUmzJd1WzvNxkoaWut0kXdoUx39Lek/5voakk0tMt0n6naR3SOopx39/7fzOkPTKzv/6EREREau+JABLW2h7nO2tgN2BdwJfbNHuOOCXtrexvSUwGUDSMOBy4CTbrwW2AXYEPlnre5HtccBbgKObLlJPL/sfA1wEXCtpo1r9gaV+nO39mvsB+wDfkrRGq4Oz/RQwCfhGifdVwEeBo+vtSsJyKXCN7c3LMX4BeJmk4cBlwJdtbwGMA3aR9K+1IR4AjqK9nW1vDbwZ2KIRTwdOBF4KbFn6vwdYx/b4cvzHAefXztH/djhuRERERFdIAtAL238FJgKHNu7g14ykushttJ1Vvn4Q+LXta0r5U8ChlAShafxHgHvKWK32fxFwTRmz05jnAk8B6/fS5krgUUkHAlOAL9he0NRsd+AJ2+fU+t1i+9fAh4Hrbf9PKX8S+PemY7wU2K7xxKCXWB6jOsfvlzSit7aS1gEmAIfZfrb0/7Pti3vrFxEREREvSALQB9t/oDpPGzdVnQ18R9J1ko6W9IpSvhUwvWmMe4G1Ja1bLy9339cCZtHeLcDratvn16a3nNLcWNJ2wNySvPTm08ApVHfPL2xRv3XzcdS0Osa7gA3L0wGAxWX8z/cRByX5+CPQa7IAjAHus/1EX2O2I2limS7Us+ip5pwnIiIiYtWXtwB1pvnuP7avlrQpsCewF3CrpK1LW7cZp1G+v6Rdqaa+fNz20/3Y94G2e1q0myTp40Ajpl7Z/l9J1wPLcve8t2Os+wHw+ZLodDImvYzbyf76ZHsqMBVg6MgxAzJmRERExMokTwD6UC7yFwFL3VG3/ajtC2x/GLgZeCswBxjfYownbD9eii4qawx2Bk6T9PJeQtgWuKODUE8v8/H3B86TtFYHfRaXTytzgO17qWs+xtcCj5QpTwDY/jtwOvDZ3oIoU39eCcwFHmHp6UsbAA+X+tdIeklv40VEREREe0kAelEW334TOMu2m+re1pjuUuambwbcD5wP7CRpt1I3DDgTOLl5fNvTqO6Sf7rN/t8H7AG0mqLTku1LgB7g4E77tHENsK6kj9bi2UHSzlQx71qeYlDOQ8tjBL5D9YRkg1Y7Kefuv4Afl/UAdwKvLgkFkl5DNeVoVkmgzgOmNBY5lzcVHbicxxoRERHRNZIALG1YmV8/B/gV1YXwl1q02x7okTQLmAacY/tm2wup3sRzjKS7gNlUTweWes1m8VXgI+VCGKqpPDMkzQU+BLzN9kO19vU1AL9qM+ZxwPOvL6V6K9AD5TOtk5NQEp59gHdKulfSbcAxwJ/Kot/3AMeWY5wF3ESVLDWP8wzVeomNmqpulDQb+C1wL+UtSWU61EHADyTNoHoT0kdrT08mAwuAO0r/S2jxdCYiIiIiWlPTje2IrjF05BiPPHhKv/rkl4AjIiLixSBpuu3xfbfsWxYBR9caO2oEPbmgj4iIiC6TKUAREREREV0kCUBERERERBdJAhARERER0UWSAEREREREdJEkABERERERXSQJQEREREREF0kCEBERERHRRZIARERERER0kSQAERERERFdJL8EHF1r9vwFjJ58Zb/6zMsvB0dERMRKLk8AVkKSFkmaIWmOpJmSDpe0WqnbRdICSbdKulPSqbV+x0o6ommseZJeWr5b0mm1uiMkHdvcV9Jakn4p6YtN8TQ+k0v59ZJ6auONl3R9LU5Lenet/gpJu9T63lUb8+JSvkWpmyHpDklTS/lwSedLmi3pNkk3SVp7wE56RERExCoiTwBWTgttjwOQtDFwATAC+GKpv9H23pKGAbdK+qntX3cw7jPAeyWdaPvhVg0krQn8BJhu+0vN8bSwsaS9bF/Vou4B4GjgZ236Hmi7p6nsTOB025eVeMaW8k8Df7E9tpRvAfy9zbgRERERXStPAFZytv8KTAQOlaSmuoXADGBUh8M9B0wFJrWpXx34ITDX9uQOxzwFOKZN3UxggaTdOxwLYCRV4gCA7dm18vm18rtsP9OPcSMiIiK6QhKAVYDtP1D9LTeul0taHxgD3NCP4c4GDpQ0okXdZ4HnbH+mqXxY0xSg/Wt104BnJO3aZn/H0z5BOL825iml7HTgWklXSZokab1S/l3gc5KmSTpe0pg+jzQiIiKiC3WUAEgaMtiBxHKr3/3fWdIs4EHgCtsPlnK36ft8ue3HgPOAw1q0uwl4s6TXNpUvtD2u9rmoqb7tRb7tGwEk7dyi+sDamEeW9t8DXg/8GNgF+K2kobZnAJtSPXHYALhZ0uubB5Q0UVKPpJ5FTy1oFVJERETEKq3TJwD3SDpF0paDGk0sE0mbAouAv5aiG22/ARgL/Jukxvz8R4D1m7qvA/ytqWwKcAjwkqbyG4DPAFdJekWn8dm+FlgLeFObJidQrQXodLw/2f6u7X2opi1tXcqfsH2J7U8C/w28s0XfqbbH2x4/ZHirhxwRERERq7ZOE4A3AHcD50j6bbmLuu4gxhUdkrQR8E3gLNtL3OG3fTdwIvC5UnQD8M+S1il93wvMtL2oqd+jwI+okgCa6n5CdZf9F7XpN504gWoK0VJsX0OVmGzT1yCS9pS0Rvn+cmBDYL6kt5QpT42FylsCf+xHfBERERFdoaO3ANl+HPg28G1JbwUuBE4vr2b8su17BjHGWNowSTOANajugP8A+Fqbtt8EjpD0GtuzJJ0F3CTJVE8MPtam32nAoa0qbH+zXHxfLmmPWjwNv2heJGz755Ie6uWYTgAuayo7X9LC8v1h27sBewBnSHq6lB9p+8ESx3+VhdCrAVdSva0oIiIiImrUdNO4daNqDcC7gI8Ao6kuOM8Hdga+Yrt5TnjEP7yhI8d45MFT+tUnPwQWERERLwZJ022PH4ixOv0dgLnAdcAptn9TK7+4PBGIWOmMHTWCnlzQR0RERJfpMwEod//PtX1cq3rbrd4WExERERER/4D6XARcFoi2e4d7RERERESsRDqdAvSbsnj0IuDJRqHtWwYlqoiIiIiIGBSdJgA7ln/r04AMvG1gw4mIiIiIiMHU6WtAMwUoIiIiImIV0NEPgUkaIelrknrK5zRJ+RnViIiIiIiVTKe/BPxd4HHg/eXzGPC9wQoqIiIiIiIGR6drADaz/b7a9peafvk1IiIiIiJWAp0+AVgoaafGhqS3AAsHJ6SIiIiIiBgsnT4B+Dfg+2Xev4BHgQmDFVRERERERAyOTt8CNAPYRtK6ZfuxQY0qYgWYPX8Boydf2e9+80561yBEExEREbFidPoWoMMlHQ58DPhY2T5E0rjBDW/5STpa0hxJsyTNkLRDKb9e0l2lbIaki0v5mZK+0NT/7DZjT5R0Z/n8vmmaVPP4+5XyRbWyGZJGS9pF0oKm8t1Ke0s6rTbuEZKOLd+3KPuZIekOSVNLeW/jtTwfvZy/YyXNr41zUilfU9IUSfdKmivpMkmb1Po1jvM2ST+TtF6t7rWSfi7pnhL3jyS9rMR9RWkzQdJDZYw5ki6WNLwW0xEtYm0+t5N7O7aIiIiIbtTpFKDx5fOzsv0u4GbgE5J+bPvkwQhueUl6M7A3sJ3tZyS9FFiz1uRA2z1N3Y4BZkg6n+rHzj4GbNti7L2BfwV2sv2wpO2ASyW90faDvYy/0Pa4prFGAzfa3rvFYTwDvFfSibYfbqo7Ezjd9mVlnLG1uqXG6+B8tHO67VObyr4CrAO81vYiSR8BLpG0g23Xj1PS94FPASdIWgu4Ejjc9s9K/a7ARi32e5HtQ0ubC4D96f3tU0ud24iIiIhYUqeLgDekumj8D9v/QZUMbAS8lX/stQAjgYdtPwNg+2Hbf+qtQ5nedDRwFnA28J+2/9ai6eeAIxsX5bZvARoXugPpOWAqMKlF3UjggcaG7dl9jNXv89FKuRP/EWCS7UVlrO9RJSutfh16GjCqfP8gMK1x8V/6Xmf7tl72tzrwEuD/9TfWiIiIiFhSpwnAq4Bna9t/B15teyHVRd8/qmuAV0q6W9I3JP1TU/35tekipzQKbV8IrA+sa/sHbcbeCpjeVNZTyluNv2EpG1Yr+2mt7c5N01c2q9WdDRyopX987XTgWklXSZpUn2bTZry+zkc7k2rjvAPYHLi/xVqQ5uNH0hDg7cDlpWhrlj5v7eyv6nWz84ENeOEJVDvDmo55/w73ExEREdE1Op0CdAHwW0mXle13AxdKeglw+6BENgBsPyFpe2BnYFfgIkmTbZ9bmrSaokOZy/5ywJLWtv1Eh7sU1bShho6mABXtpgBh+zFJ5wGHUXv9qu3vSboa2BPYB/hXSdv0Nl4f56OdJaYAlX24Rbv68Q8rF++jqS74f9nHPlq5yPahkkSVBB0JnNRL+z6nAEmaCEwEGLJuq1lHEREREau2jp4A2P4y8HHgb8AC4BO2j7P9pO0DBzPA5WV7ke3rbX8ROBR4X199gDOAY4EfAV9s0+Z2YPumsu0YvIRoCnAI1VSY59n+k+3v2t6HarrQ1r0Nsozno9k9wKslrdNUXj/+xsX4q6nWGTSmRs1h6fPWq7Km4GdUU86Wi+2ptsfbHj9kePMDlYiIiIhVX6dTgACGAY/ZngL8UdJrBimmAVPekjOmVjQO+GMfffYCNgbOA74M7CtpyxZNTwa+2pjao+qNSBOAbwxA6Eux/ShVQnJILdY9Ja1Rvr+caq3G/HZj9HY+JJ0oad8OY3mSar3D18oUHyQdBAwHrm1qu4DqycURJdYLgB0lPf8uzXIc9QXMrewE3NtJfBERERHRXkdTgCR9kWrh7xZUb2FZA/hv4C2DF9qAWBv4epkb/xzVneuJtfrzJTWm1DxM9YacKcB+5a7zk5I+S7UgeInFrbYvlzQK+I0kA48DH7L952WMdecyZabheNsXN7U5jequfcMewBmSni7bR9p+UNLrWo0H3Ef78zGWF+bpd+LzwKnA3ZIWA3cC+5bztgTbt0qaCXzA9g/KG5SmSJpCtZ5kFvBpqgSmbn9Vr1ZdjWqx84Ra3TGSPlPbxya8MO2o4Re28yrQiIiIiBq1uF5bulF1UbUtcIvtbUvZLNtvGOT4YgWRdLXtd7zYcaxIQ0eO8ciDp/S7X34ILCIiIlY0SdNtjx+IsTpdBPysbZc73ZTFv7EK6baLf4Cxo0bQk4v5iIiI6DKdrgH4kaRvAetJ+jjwK+CcwQsrIiIiIiIGQ0dPAGyfKml34DGqdQD/aXtZXusYEREREREvok4XAX/V9ueovcu9VhYRERERESuJTqcA7d6ibK+BDCQiIiIiIgZfr08AJP0b8ElgU0mzalXrAL8ezMAiIiIiImLg9TUF6ALgKuBEoP4+9cfLD1NFRERERMRKpNcEoPyK6wLgAABJGwNrAWtLWtv2/YMfYkREREREDJSO1gBIerekuVS/JPt/gXlUTwYiIiIiImIl0uki4OOBNwF3234N8HayBiAiIiIiYqXTaQLwd9uPAKtJWs32dcC4QYwrIiIiIiIGQUe/AwD8TdLawA3A+ZL+Cjw3eGFFDL7Z8xcwevKV/e4376R3DUI0EREREStGr08AJG0u6S3APsBTwCTgF8AjwL8PfnirFklHS5ojaZakGZJ2kPTT8v0eSQvK9xmSdpS0pqQpku6VNFfSZZI2qY33RIt9HCtpfm2cGZLWk7SLJEt6d63tFZJ2aTHGuZLuK31vkfTmFuUzJb291qevWBeVfrdJ+rGk4U3ljc/kUn69pLvKfm6WNK6U/660u1/SQ7V+F5TX1jb2t0M5z50muRERERFdoa+LoynAUbafLNuLge9LGg8cC7y7XcdYUrmI3hvYzvYzkl4KrGl731K/C3CE7b1rfU6l+s2F19peJOkjwCWSdrDtXnZ3uu1Tm/YP8ABwNPCzDkI+0vbFkvYAvgW8oal8V2AqMKaUf6WPWBfablzEnw98AvhavbyFA233lLFOAXa3vUMZYwIw3vahZftlwDRJF1MlqGcBn7SdJ1URERERNX2tARhte1Zzoe0eYPSgRLTqGgk8bPsZANsP2/5Tu8blDvlHgEm2F5U+3wOeAd62jDHMBBZIavXLzu3cAGzeonwaMGoZY72xzZjtPL+vdmz/BTgVOJkquZhl+6Z+7CMiIiKiK/SVAKzVS92wgQykC1wDvFLS3ZK+Iemf+mi/OXC/7ceaynuArfroO6k2Nea6prrjgWM6D5t3A7NblO8JXNrfWMuUnL1qYw5rmgK0fx/76s03gS2BI4HPdtA+IiIiouv0NQXoZkkft/3teqGkQ4DpgxfWqsf2E5K2B3YGdgUukjTZ9rltughoNc2nXXndUlOAanHcKAlJO/cxximSjgEeAg5pKj8Z2Jjq1bCdxjpM0ozy/UbgO+V7b1OAzpf0EmAIsF0f8WJ7saRvUU0NeqRVG0kTgYkAQ9bdqK8hIyIiIlY5fSUAnwF+KulAXrjgHw+sCew7mIGtisr0mOuB6yXNBg4Gzm3T/B7g1ZLWsf14rXw7OpvD35sTqNYC9DY//kjbF7cqBy4BDgO+D2zfYay9Xei3cyDVtKWTgLOB93bQZ3H5tGR7KtXaBYaOHNNXIhURERGxyul1CpDtv9jeEfgS1a//zgO+ZPvNth8c/PBWHZK2kDSmVjQO+GO79mXh9feBr0kaUsY4CBgOXLs8sdi+Blgf2GYZ+y8GzqD6XYh3DHKsf6easvQmSa9fnrEiIiIiosPfASg//NU8lzz6Z23g65LWo7rzfg9lKkovPk+1sPVuSYuBO4F9a28AGi7pgVr7r5V/J0n6UK38PS3GPgG4rJ/H8DzblnQ81Vz7qzuItZ361CCAX9ie3LSvhZJOA45gyelIEREREdFP6vv6LGLVNHTkGI88eEq/++WHwCIiImJFkzTd9viBGCs/khRda+yoEfTkYj4iIiK6TF+vAY2IiIiIiFVIEoCIiIiIiC6SBCAiIiIiooskAYiIiIiI6CJJACIiIiIiukgSgIiIiIiILpIEICIiIiKiiyQBiIiIiIjoIkkAIiIiIiK6SBKAiIiIiIgusvqLHUDEi2X2/AWMnnzlgI8776R3DfiYEREREQMlTwBWcpIWSZoh6TZJP5Y0vEX5zyStV+uzlaRrJd0taa6kL0hSqXuZpCskzZR0u6SfSxpbxpoh6VFJ95Xvv5I0uuzjHbU2T0i6q3w/T9IESWc1xX29pPGSflfa3S/podoYoyXNk/TS0n4TSZeVeO+VdIakNUvdLpIs6d218a+QtMsK+BNERERErFSSAKz8FtoeZ3tr4FngEy3KHwU+BSBpGHA5cJLt1wLbADsCnyz9jgN+aXsb21sCk23PLmONK32PLNu7NYKwfXWtTQ9wYNk+qLfgbe9Q+vwncFFjDNvzGm1KcnIJcKntMcBrgbWBE2pDPQAc3c9zFxEREdF1kgCsWm4ENm9RPg0YVb5/EPi17WsAbD8FHApMLvUjqS6mKfWzBi3azr0NeNr29wBsLwImAR9tPPEAZgILJO3+IsUYERERsVJIArCKkLQ6sBcwu6l8CPB2qjv3AFsB0+ttbN8LrC1pXeBs4DuSrpN0tKRXDHrwfWsV82PA/SyZ8BwPHLMC44qIiIhY6SQBWPkNkzSDatrN/cB3msofATYAflnKBbjNWLZ9NbAp8G3gdcCtkjZazhjb7q/D/u1iXqLc9o0AknZuO5A0UVKPpJ5FTy3ocPcRERERq44kACu/hbV58/9u+9l6OfBqYE3KGgBgDjC+PoCkTYEnbD8OYPtR2xfY/jBwM/DW5YzxEWD9prINgIc77N8q5nWBVwL3NrU9gV7WAtieanu87fFDho/ocPcRERERq44kAKs42wuAw4AjJK0BnA/sJGk3eH5R8JnAyWX7bbU3Ca0DbEb1ZGF53Ay8RdLLy7jjgaHA/3bY/3+A4ZIOKv2HAKcB55Y1DM8raxvWp1rcHBERERFNkgB0Adu3Ui2S/YDthcA+wDGS7qJaM3Az0HhN5/ZAj6RZVIuHz7F983Lu/y/Ap4Gfl2lJU4ADbC/usL+BfYF/kTQXuBt4GjiqTZcTgE2WJ+aIiIiIVZWqa6uI7jN05BiPPHjKgI+bHwKLiIiIgSZpuu3xfbfsW34JOLrW2FEj6MnFekRERHSZTAGKiIiIiOgiSQAiIiIiIrpIEoCIiIiIiC6SBCAiIiIiooskAYiIiIiI6CJJACIiIiIiukgSgIiIiIiILpIEICIiIiKiiyQBiIiIiIjoIkkAIiIiIiK6yOovdgARL5bZ8xcwevKVAz7uvJPeNeBjRkRERAyUPAGIliRtKGlG+TwoaX5t2+Xf2yT9TNJ6pc9oSQtr7WZIOqjUfVTSbEmzSr99Svm5ku4rbW+R9OZaDKtLeljSiU2x7S3pVkkzJd0u6V9L+RaSri9j3SFp6oo7YxERERErhzwBiJZsPwKMA5B0LPCE7VPL9hO2G3XfBz4FnFC63tuoa5C0CXA0sJ3tBZLWBjaqNTnS9sWS9gC+BbyhlO8B3AW8X9JRti1pDWAq8EbbD0gaCowu7c8ETrd9Wdnv2AE6HRERERGrjDwBiOU1DRjVR5uNgceBJwBsP2H7vhbtbgA2r20fAJwB3A+8qZStQ5W4PlLGesb2XaVuJPBAo7Pt2f06koiIiIgukAQglpmkIcDbgctrxZs1TQHaGZgJ/AW4T9L3JL27zZDvBmaXsYeVsa8ALqRKBrD9aNnfHyVdKOlASY3/x6cD10q6StKkxtSkiIiIiHhBEoBYFsMkzaC6C78B8Mta3b22x9U+N9peBOwJ7AfcDZxephU1nFLGmwgcUsr2Bq6z/RTwE2DfknBg+2NUycHvgSOA75by7wGvB34M7AL8tkwRep6kiZJ6JPUsemrBAJ2OiIiIiJVHEoBYFgvLPP9XA2tSrQHolSu/t30i8AHgfbXqI0uysLvt20rZAcBukuYB04ENgV1r4822fTqwe30s23+y/V3b+wDPAVs3xTHV9njb44cMH9H/I4+IiIhYySUBiGVmewFwGHBEWZzbkqRXSNquVjQO+GMv7dcFdgJeZXu07dFUScYBktaWtEursSTt2YhD0supkob5y3JsEREREauqvAUolovtWyXNpLqrfyNlDUCtyXeBy4BTJb0CeBp4CPhEL8O+F7jW9jO1ssuAk4HDgc9K+hawEHgSmFDa7AGcIenpsn2k7QeX5/giIiIiVjWy/WLHEPGiGDpyjEcePGXAx80PgUVERMRAkzTd9viBGCtPAKJrjR01gp5crEdERESXyRqAiIiIiIgukgQgIiIiIqKLJAGIiIiIiOgiSQAiIiIiIrpIEoCIiIiIiC6SBCAiIiIiooskAYiIiIiI6CJJACIiIiIiukgSgIiIiIiILpIEICIiIiKii6z+YgcQ8WKZPX8BoydfOej7mXfSuwZ9HxERERGdyhOAlZCk0yV9prZ9taRzatunSTpc0mhJCyXNqH0OqrXbVpIlvaNp/EWl7W2SfixpeCnfRNJlkuZKulfSGZLWLHW7lLHeXRvnCkm7tIj/XEn7tTm2SZKeljSiVnarpHHl++qSnpT0oVr9dEnbSXpZ2edMSbdL+nm/TmxEREREF0gCsHL6DbAjgKTVgJcCW9XqdwR+Xb7fa3tc7XNerd0BwE3l37qFpe3WwLPAJyQJuAS41PYY4LXA2sAJtX4PAEcv57EdANwM7Fsre/54gW2Au3jh+F8CbArMBI4Dfml7G9tbApOXM5aIiIiIVU4SgJXTr3nhgngr4DbgcUnrSxoKvB64tbcBygX9fsAEYA9Ja7VpeiOwOfA24Gnb3wOwvQiYBHy08YSA6iJ8gaTdl+WgJG1GlVQcw5JJSf14dwS+CYwr228EbinxjKRKQigxzlqWOCIiIiJWZUkAVkK2/wQ8J+lVVBfE04DfAW8GxgOzbD9bmm/WNAVo51L+FuA+2/cC1wPvbN6PpNWBvYDZVInG9KY4HgPup0oQGo6nUuog1wAAIABJREFUuoBfFgcAF1IlHVtI2riU158A7AjcADwjaR2WfNpxNvAdSddJOlrSK5YxjoiIiIhVVhKAlVfjrngjAZhW2/5NrV3zFKAbS/kBwA/L9x+y5B33YZJmAD1UF/jfAQS4RRxLlDfGryUa/fEB4Ie2F1NNN/qXMuY8YE1JLwdeRzUF6GZgh/rx2r6aajrQt0u7WyVttESw0kRJPZJ6Fj21YBlCjIiIiFi55S1AK6/GXfGxVFOA/hf4D+Ax4Lu9dZQ0BHgf8M+Sjqa6iN9Q0jq2H6esAWjqM6f0qZetC7wSuBfYsFZ1AtVagOc6PRhJbwDGAL+sZiexJvAHqrv6UCU4+wF/tm1Jv6V6ivFG4LeNcWw/ClwAXCDpCuCtwE9q9VOBqQBDR45pldBERERErNLyBGDl9Wtgb+BR24vKhe96VNOApvXRdzdgpu1X2h5t+9VUF8nv6aXP/wDDG28RKknEacC5tp+qN7R9DbA+1YLdTh0AHFviGW37FcAoSa+uHe+k2rFNAw4CHrT9txLT22pvLFoH2IzqCUZEREREFEkAVl6zqd7+89umsgW2H66VNa8BOIzqYvunTeP9BPhgu53ZNtWbef5F0lzgbuBp4Kg2XU4ANukl/m9JeqB8plFN/2mO6aelHKoEYFNKAmD7z8AQlpzutD3QI2lWaXeO7Zt7iSEiIiKi66i6rovoPkNHjvHIg6cM+n7yQ2ARERGxvCRNtz1+IMbKGoDoWmNHjaAnF+cRERHRZTIFKCIiIiKiiyQBiIiIiIjoIkkAIiIiIiK6SBKAiIiIiIgukgQgIiIiIqKLJAGIiIiIiOgiSQAiIiIiIrpIEoCIiIiIiC6SBCAiIiIioovkl4Cja82ev4DRk69c4fudl18fjoiIiBdRngBERERERHSRJACDQNLRkuZImiVphqQdSvn1ku4qZTMkXVzrc5Ck20q/2yUdUcol6RhJcyXdLek6SVvV+s2T9JPa9n6Szi3fJ0h6SNKtpf/VknZsinV1SQ9LOrGpvBHrTEk3SxpXyi+Q9G+1djuU41y9qf8akk4q+71N0u8l7VXqRkg6T9K95XOepBGlbrSkhSXmO0q/g2vjNo5pRu2zpaTVJJ1Z9jW7xPyaZf4jRkRERKyiMgVogEl6M7A3sJ3tZyS9FFiz1uRA2z1NffYCPgPsYftPktYCPlyqPwXsCGxj+ylJewCXS9rK9tOlzfiyPadFSBfZPrTsZ1fgEkm72r6j1O8B3AW8X9JRtt0cq6SPAKcAuwOTgGkleXkEOAv4pO3nmvb7ZWAksHU5Dy8D/qnUfQe4zfZBJa4vAecA/1Lq77W9banbtMS8mu3vNR9T7RweALwCeIPtxZI2AZ5scT4iIiIiulqeAAy8kcDDtp8BsP2w7T/10efzwBGNdraftv3tUvc54N9tP1XqrgF+AxxY638qcFRfgdm+DpgKTKwVHwCcAdwPvKlN12nAqDLGX8r+TgY+AcyyfVO9saThwMdL3I3z8BfbP5K0ObA9VYLQcBxVErNZi5j/ABwOHNbH4Y0E/mx7cen3gO3/10efiIiIiK6TBGDgXQO8skzX+Yakf2qqP782deWUUrY1ML15IEnrAi+xfW9TVQ+wVW37R8B25eK6L7cAryvjDwPeDlwBXEiVDLSyJ3BpbfubwJbAkcBnW7TfHLjf9mMt6rYEZthe1Cgo32c0HVPLmIv9m6YADaM6B+8u26dJ2rbNWBERERFdLQnAALP9BNUd7onAQ8BFkibUmhxoe1z5HLmMuxFQn6qziGqKzuc77NuwN3BdebrwE2BfSUNq9edLeoDqKcTXG4XlLvu3gKtsP7KcsfdV3hwzVFOAxtU+C20/AGxBdQ4WA/8j6e1LDSRNlNQjqWfRUwv6GXpERETEyi8JwCCwvcj29ba/CBwKvK+PLnOokobmcR4Dnizz4Ou2A25vKvsB8FbgVX3sa1ugMf//AGA3SfOonkBsCOxaa3sg8BrgAuDspnEWl08r9wCvkrROi7o5wLaSnv+/V75vU4urt5jbsv2M7atKYvUV4D0t2ky1Pd72+CHDR/Q1ZERERMQqJwnAAJO0haQxtaJxwB/76HYicLKkl5cxhkpqzHk/BTizTHNB0m7ATlQX5c+z/XfgdKrFxO1i+yeqJxPfLtOLdgJeZXu07dFUC46XmAZUxj0GeJOk1/dxHI0+T1Et9D1T0ppl3yMlfcj2PcCtZcyGY4BbSl1zzKOp1hx8vbmuqd12kl5Rvq8GvIG+z3tERERE18lbgAbe2sDXJa0HPEd1N7y+6PZ8SQvL94dt72b75+UtOb+S1JgK893S5uvA+sBsSYuAB4F9bC9kad9hyQtrqObL7wQMB+4D3mf7jjIt6drGIt3iMqpEZGh9ANsLJZ0GHAEc0uF5OAY4Hrhd0tNUb+T5z1J3CNU5uodqes+0pnE3k3QrsBbwOPD12huA6sfU8ElgXarEphH776neUBQRERERNVryrY8R3WPoyDEeefCUFb7f/BJwRERE9Jek6bbHD8RYeQIQXWvsqBH05GI8IiIiukzWAEREREREdJEkABERERERXSQJQEREREREF0kCEBERERHRRZIARERERER0kSQAERERERFdJAlAREREREQXSQIQEREREdFFkgBERERERHSR/BJwdK3Z8xcwevKVK3y/8/LrwxEREfEiyhOAiIiIiIguslInAJIWSZpR+4wu5TtJ+r2kO8tnYq3PsZLml/a3SzqgaczDS5/ZkmZK+pqkNUrdvFLe2N+ZpfxcSfeVspmS3l7KvyLpq7WxXy3pD5LWK9sHSDq6fN9LUo+kO8r+T631m1g7lt9L2qlWN0/SS2vbu0i6onyfIGmxpDfU6m+TNFrS70q890t6qH4O+3ucpe56SXfV+uzX5m80udZ+fAdxvLS0t6TTavs7QtKxte0PSZolaU6J7ZzGeY6IiIiIF6zsU4AW2h5XL5D0cuAC4D22bykXkFdLmm+7Md/jdNunShoDTJd0se2/S/oEsAfwJtt/k7QmcDgwDPh76bur7YdbxHKk7Ysl7QpMBcYAXwZulXSu7TuAM4Av2P5b6bMncKakrYGzgHfZvlPS6sDEcjx7A/8K7GT7YUnbAZdKeqPtBzs4Rw8ARwP71wtt71DGnwCMt31o7Rz29zgbDrTd09R+qb9RP+NoeAZ4r6QTm+OStCcwCdjL9nxJQ4CDgZcBfyMiIiIinrdSPwFo41PAubZvASgXi58FJjc3tD0XeApYvxQdDfxb4wLd9rO2T7L9WD/2Pw0YVfovpEogviFpL2Ad2+cDqLq6HQfcUuI7wfadpd9ztr9Rxvsc1UX3w6XuFuD75Tg7cQWwlaQt+nEMnXj+OFeQ56gSjkkt6o4GjrA9H8D2ItvftX3XCowvIiIiYqWwsicAw2pTRn5ayrYCpje16ynlSyh30+fa/qukdYC1bd/Xxz6vq+2z1cXonsCljQ3bPwceBc4DPllrty0w07aBrVvE3NDx8bSxGDgZOKrD9g39Os7i/FqfDUtZ/W80Q9L+LLuzgQMljWgq34oqkYqIiIiIPqxyU4AAAW7Rtl42SdLHgU2pLmSX6ifpHcBXgfWAD9r+TalqNzXmFEknAxsDb2qqOxsY1nRHek/gqrZH1rt6rH0dK1RToo6W9Jp+7GNZjrPfU4D6w/Zjks4DDgMWtmojaSzwA2Ad4CjbFzXVT6RMrxqy7kYDEVZERETESmVlfwLQyhxgfFPZ9sDtte3TbW9BNS/+PElrlWk+TzYukm1fXS5cbwPW7GC/RwKbA8dQTdGpW1w+dXsA19Ri3r7NuLe3qNuudjyP8MIUJoANgCUu3G0/B5xGNZ1oefV2nCvCFOAQ4CW1sjlU5wTbs8vf7SqqtRtLsD3V9njb44cMb36QEBEREbHqWxUTgLOBCZLGAZSpKF+lmgazBNuXUE2nObgUnQj8V+0tPQLW6nTHthdTLfRdrTxBaKlMYVnd9iOl6BTgKEmvLfWrSTq81J0MfLUxpaYc1wSgsUbgeuDDpW4I8CHguha7PRfYDVju296dHudgsP0o8COqJKDhROBUSZvUypa6+I+IiIiIlX8K0FJs/1nSh4Bvl3n9AqbY/lmbLscBF0j6NvBfwHDgd5KeAZ4Afg3cWmt/naRF5fss2wc17d+Sjqda2Ht1m33uDvyq1meWpM8AF0oaTjWF58pSd7mkUcBvJBl4HPiQ7T+X7l+mSlpmlmP9BfDfLc7Ls+V1nme0ianZQBwnlDUAte1f2F5qQXY/nQY8/7Yg2z+XtBFwVUmC/kb15Ka3uCIiIiK6kqo1qLEiSToHOMf2b1/sWLrZ0JFjPPLgKSt8v/kl4IiIiOgvSdNtN09zX7axkgBEtxo/frx7eprXLEdERET84xnIBGBVXAMQERERERFtJAGIiIiIiOgiSQAiIiIiIrpIEoCIiIiIiC6SBCAiIiIiooskAYiIiIiI6CJJACIiIiIiukgSgIiIiIiILpIEICIiIiKii6z+YgcQ8WKZPX8Boydf+WKHsdzmnfSuFzuEiIj4/+3de7geVX33//fHIAHkIIpoADWiKVoOhpjHWgGLZ6y0SMVCRIVqi8e24gOKh/5KrRZUKGBRLLaKWAStZ8FjBR5BqZBAIAE5SrTEE4caRRAlfH9/3GvLcGefAoGdvef9uq772jNrrVlrzdqzk/nOrJlbmka8AyBJkiT1iAGA7jdJVidZ2vnMTbJnklVJLklyZZJjRtnuC0kuGEo7Jcl+Q2m3tp9zk9ze6vxekguTHHT/7p0kSdL05BQg3Z9ur6r53YQkc4HzqmrvJBsDlyT5XFV9u+U/FFgA3JrkcVV1/STbuq6qdm11bA98NsmDquqj62pnJEmSZgLvAGjKVNXtwFJg207yi4EvAWcAB9zLer8PvAn4m/vaR0mSpJnGAED3p407038+N5yZZEtgHvCtTvIi4PT2WXQf2r4YeOIobR6SZHGSxatvW3UfqpckSZqenAKk+9MaU4CaPZJcBuwAHF1VPwFI8kjgCcD5VVVJ7kyyU1UtB2qUekZLG5HREqvqZOBkgNlz5o23vSRJ0ozkHQBNhfOqahdgZ+C1SUaChP2BLYHrk6wA5nL3NKCbWx4ASR4G3DROG7sC31u33ZYkSZr+DAA0ZarqauAo4C0taRGwV1XNraq5wFO4OwA4F9g/yYZt/WDgnNHqbQ8aHwP8y/3QbUmSpGnNKUCaah8CDkvyOOAxwH+PZFTV9Ul+keQPqurMJE8BliRZDVwHvKZTz+OTXAJsBPwS+BffACRJkrQmAwDdb6pq01HSzmVwNX9k/XbufgvQtqOUX9BZ/gfgH0YpswLY+L72V5IkqQ8MANRbO2+7BYuPfuFUd0OSJOkB5TMAkiRJUo8YAEiSJEk9YgAgSZIk9YgBgCRJktQjBgCSJElSjxgASJIkST1iACBJkiT1iAGAJEmS1CMGAJIkSVKP+E3A6q1lK1cx94izprob640VfiuyJEm94B0ASZIkqUcMADQpSVYnWZpkeZIvJXloJ2/HJGcnuTrJNUn+Lkla3sFJbmzbXp7k00k2aXlHJlnZ8q5MclKSB7W8U5Jc3/KWJvnOKPVdmeTQlv68JBd02p3Vyjz9gR4rSZKk9ZkBgCbr9qqaX1U7AbcArwdIsjHwReDoqvo94MnA04HXdbb9ZNt2R+A3wP6dvOOqaj7w+8DOwB918g5v282vqqcP1wfsBrw9yaOr6uvAD4BXtTJ/DVxUVd9ZN7svSZI0M/gMgO6NC4Bd2vJLgW+3E3Cq6rYkbwDOBT7Q3SjJBsBDgP8dpc4NgY3GyBtVVd2c5FpgDvA/wKHA+UkuAN4APHUt9kmSJKkXvAOgtZJkFvBsBlf9AXYElnTLVNV1wKZJNm9J+ydZCqwEHgZ8qVP80Jb3Y+DqqlrayXtfZwrQaaP05TEMgobLWrs/Bo5nEKC8q6puGWWbQ5IsTrJ49W2r1nb3JUmSpj0DAE3Wxu1E/WYGJ/HfaOkBaoxtRtJHpuw8ClgGHN4pMzIFaGvgIUkO6OR1pwAd2EnfP8nlwPeBE6rq1528DwCzquqUUTtUdXJVLayqhbM22WKifZYkSZpxDAA0Wbe3E/XHMpiu8/qWfjmwsFswyfbArVX1y256VRWDq//PGK68qn4LfHW0vFF8sj1PsAdwbJJHdeq5i7EDEkmSpN4zANBaqapVwN8AhyV5MHAasHuS58DvHgp+P/DeMarYHbhuOLG9vefpo+WN05cLgI8Df7s2+yBJktRnBgBaa1V1CXApcEBV3Q7sA7wjyVUMpvhcBJzY2WT/No//MmBX4B87eSPPACxn8FD6Bzt53WcAlibZcJTuvAf4iySbrbMdlCRJmsEymJUh9c/sOfNqzkHHT3U31ht+E7AkSeuvJEuqauHEJSfma0DVWztvuwWLPemVJEk94xQgSZIkqUcMACRJkqQeMQCQJEmSesQAQJIkSeoRAwBJkiSpRwwAJEmSpB4xAJAkSZJ6xABAkiRJ6hEDAEmSJKlH/CZg9daylauYe8RZU92NaWWF35wsSdK05x0ASZIkqUcMANYzSSrJxzvrGyS5McmZbf3gJCe25R2SnJtkaZLvJTm5pW+S5LQky5IsT3J+kk1b3q3t59zW1l932joxycGd9TclubLVc2mSf07y4FH6fG6Sq1qZbyfZYSh9aft8uqUfmWRlS7siyaJOXU9L8t3OPh3ZyXtRkss6fXpRJ++UVufstr5VkhX36ZchSZI0AzkFaP3zK2CnJBtX1e3Ac4GVY5R9P3BcVX0BIMnOLf1vgZ9W1c4tfQfgt6Ns/zPgb5P8a1X9ppuR5DXA84CnVdXPk2wIvAnYeIy6DqyqxUkOAd4H/Gk3fZTyx1XVMUnmAUuSfLqqfgt8DPjzqro0ySxgJJh4MnAM8Nyquj7J44BvJPl+VV3W6lwNvBI4aYzxkiRJ6j3vAKyfvgKMTLZeBJw+Rrk5wA0jK1W1rJO+spN+VVXdMcr2NwLfBA4aJe/twGur6uetjt9U1dFV9YsJ+v4t4AkTlPmdqroGuA3YsiVtDfy45a2uqita+mHAP1XV9S3veuAo4PBOdccDhyYxsJUkSRqDAcD66QzggCQbAbsA3x2j3HHA2Um+kuTQJA9t6R8B3pLkgiTvalfZx3I08H/b1XYAkmwGbDpysr2W/gRY1lk/rTMF6H3DhZMsAK6pqp919umqJJ9L8uo2BgA7AkuGNl/c0kf8EDgfePlYnUtySJLFSRavvm3V2u2ZJEnSDGAAsB5qU1rmMrj6/+Vxyn0UeBLwn8CewH8nmV1VS4HtGUzFeRhwUZInjVHH9cCFwEs7yQHqdyvJ89sJ/IokTx+jO6clWQrsxuBq/YgDq2p++3Sv1h+a5CoGwc2Rnf68E1gIfL316auj9WmctH9icFdg1GO7qk6uqoVVtXDWJluMsSuSJEkzlwHA+uuLDOa8jzX9B4Cq+lFVfaSq9gHuBHZq6bdW1Wer6nXAfwB/PE41/wS8hXY8tGk+v2rz7Kmqr1XVfGA5sOEYdYyc6L+oqv5nEvt3XFXtAOwPnNq50k9VXVdVJwHPBp6c5OHA5QwCg64FwBXdhKq6FlgK/Pkk+iBJktQ7BgDrr48A7+zM619Dkr1G3sqT5FHAw4GVSXZLsmVL3xD4feAHY9VTVVcyOJHeu5N8FHDSyLSiJAE2GmXz+6SqPstgKs9BrZ0XtrYA5jF4sPfnDIKhtyaZ28rNBd4GHDtKte/mnnchJEmS1Piw5Hqqqm4ATpig2POAE5L8uq0fXlU/SfI8BifvYRDknQV8ZoK63g1c0lk/CdgE+G6SO4BbgW8PlZmM05Lc3pZvqqrnjFLmncAnknyYwfz945LcxuCOxoFVtRpYmuQtwJda0PNb4M1tutM9VNXlSS5mcIdAkiRJHakankIt9cPChQtr8eLR3lAqSZK0fkmypKqGp0PfK04BkiRJknrEAECSJEnqEQMASZIkqUcMACRJkqQeMQCQJEmSesQAQJIkSeoRAwBJkiSpRwwAJEmSpB4xAJAkSZJ6xABAkiRJ6pENproD0lRZtnIVc484a6q70Tsrjn7hVHdBkqRe8w7AeijJ25NcnuSyJEuT/EFLX5Fkq065PZOc2ZYfmeTMJJcmuSLJl1v63CS3t3quSPKhJA9q6cs7dT01ybeSXJXkyiT/lmSTJAcnObFT7pCWf2WSC5Ps3slb6/6Nsu/bJflCkmuSXJfkhCQbTtTPlveCJIuTfK/lHXPffxuSJEkzi3cA1jNJ/hDYG1hQVXe0E+oNJ9gM4J3AN6rqhFbPLp2866pqfpINgLOBFwEXd9p8JPCfwAFVdUGSAC8GNhvq297Aq4Hdq+qmJAuAzyd5alX95D70b6T+AJ8FTqqqfZLMAk4G3g0cPl4/k2wPnAi8sKqubPt6yMTDJkmS1C/eAVj/zAFuqqo7AKrqpqr60SS3u2FkpaouGy5QVXcC3wGeMJT1euBjVXVBK1dV9emq+ulQubcAh1fVTa3cxcDH2vb3uX/As4BfV9VHW5nVwKHAK9tV/vH6+Wbg3VV15ci+VtUHJ9EvSZKkXjEAWP98HXh0kquTfDDJH01yuw8A/57knDaFaJvhAu0k+tnAsqGsnYAlk2hjx1HKLW7p97l/o9VfVb8AfsggaBmvn5PdB0mSpF4zAFjPVNWtwFMYTF+5EfhkkoNHskfbpG33NWB74MPAE4FLkjyilXl8kqXAt4Gzquor67DL6fTr3vZvtLomk772nR08w7A4yeLVt61aF1VKkiRNKwYA66GqWl1V51bV3wNvYDDPHeBmYMtO0YcBN3W2u6WqPlFVLwcuAp7Rsq6rqvlVtWtVHTlKk5czCDomcsUo5Ra09PvSv24/FnYTkmwOPBq4boJ+TmofqurkqlpYVQtnbbLFRMUlSZJmHAOA9UySHZLM6yTNB37Qls8FXt7KzQJeBpzT1p/VeRvOZsDjGUydmYwTgYNG3jbU6nhZkkcNlXsv8J4kD29l5gMHAyNz7e9r/74JbJLkFZ06jgVOqarbJujn+4C3Jfm9lv6gJG+a5P5LkiT1hm8BWv9sCvxLkocCdwLXcvfbbP4ROCnJpQymxXwV+I+W9xTgxCR3Mgjs/q2qLkoyd6IGq+qnSQ4AjkmyNXAX8C0Gb+Tplvtikm2B7yQp4JfAy6rqx/e2f0P1V5J9gQ8m+btW7svA2ybqZ1X9JMkbgdNboFGAL/mXJEkakqp1MrVamnZmz5lXcw46fqq70Tt+EZgkSWsvyZKqWjhxyYl5B0C9tfO2W7DYk1FJktQzPgMgSZIk9YgBgCRJktQjBgCSJElSjxgASJIkST1iACBJkiT1iAGAJEmS1CMGAJIkSVKPGABIkiRJPWIAIEmSJPWIAYAkSZLUIxtMdQekqbJs5SrmHnHWVHdDa2HF0S+c6i5IkjTteQdAv5NkdZKlSS5NcnGSp7f0PZOcOVT2lCT7teVzk1zVtrsoyfxOuU2TnJTkuiSXJFmS5K9a3twky8foywZJbkpy1FD6qG0l+USS13bK/UGSy5IY5EqSJHUYAKjr9qqaX1VPBt4KHDXRBh0Htu0+CLyvk/5vwP8C86pqV2Av4GGTqO95wFXAnyfJJNo6FDg8ySOSPAg4EXhdVd25FvsgSZI04xkAaCybMzhxX1sXANsCJHk88FTgHVV1F0BV3VhV75lEPYuAE4AfAk+bqK2q+ilwDPBe4DXAZVV1/r3ovyRJ0ozm9Ah1bZxkKbARMAd41r2oYy/g8215R+DSkZP/yUqyMfBs4NXAQxkEAxdM0BbAh4CDgD2BhWvVa0mSpJ4wAFDX7VU1Mqf+D4FTk+wE1Bjlu+mnJXkIMAtYMFrhJG8HXgJsXVXbjNOPvYFzquq2JJ8B/i7JoVW1ery2ququJP8KLKyqm8fowyHAIQCzNn/EOF2QJEmamZwCpFFV1QXAVsAjgJuBLYeKPAy4qbN+IPA44BPAB1raFcCT25x8qurdLcDYfILmFwHPSbICWAI8HHjmBG2NuKt9xtqvk6tqYVUtnLXJFhN0Q5IkaeYxANCokjyRwRX2m4FrgG2SPKnlPRZ4MrC0u01V/RZ4B/C0JE+qqmuBxcC7ksxq224EDD/U2213c2B34DFVNbeq5gKvZxAUjNnWfd9jSZKkfnAKkLpGngGAwUn6QW3azeokLwM+2k7gfwv8ZVWtGq6gqm5PcixwGPAq4C8ZvKnn2iS3ALcDb+lsskOSGzrrJwBnV9UdnbQvAO9NMnuCtiRJkjSBVI01vVua2WbPmVdzDjp+qruhteAXgUmS+irJkqpaJy858Q6AemvnbbdgsSeUkiSpZ3wGQJIkSeoRAwBJkiSpRwwAJEmSpB4xAJAkSZJ6xABAkiRJ6hEDAEmSJKlHDAAkSZKkHjEAkCRJknrEAECSJEnqEQMASZIkqUc2mOoOSFNl2cpVzD3irKnuhh4gK45+4VR3QZKk9YJ3AHouyXFJ3thZ/1qSf+usH5vkTW350CS/TrJFJ3/PJKuSXJLkyiTHdPIOTnJjkqUt79BR2r80yelDaackWZlkdlvfKsmKJDu3upYmuSXJ9W35v5I8KMn7kyxPsizJRUket25HS5IkafozANB3gKcDJHkQsBWwYyf/6cC32/Ii4CJg36E6zquqXYFdgb2T7NbJ+2RVzQd2A96e5NEjGUmexOAYfEaShwzVuRp4ZTehqpZV1fxW3xeBw9v6c4D9gW2AXapq59bHn6/FOEiSJPWCAYC+TQsAGJz4Lwd+mWTLdgX+ScAlSR4PbAq8g0EgsIaquh1YCmw7St7NwLXAnE7yS4GPA18H/nRok+OBQ5NMdpraHODHVXVXa++GqvrfSW4rSZLUGwYAPVdVPwLuTPIYBoHABcB3gT8EFgKXVdVvGJz0nw6cB+yQZOvhupJsCcwDvjVK3mOAjYDLOsn7A59s9Q4HFT8EzgdePsld+RRdq1aUAAAgAElEQVTwJ21K0LFJdp3kdpIkSb1iACC4+y7ASABwQWf9O63MAcAZ7Qr7Z4GXdLbfI8llwE+AM6vqJ528/ZNcDnwfOKGqfg2Q5P8AN1bVD4BvAgtaANH1T8DhTOI4raobgB2AtwJ3Ad9M8uzhckkOSbI4yeLVt62aqFpJkqQZxwBAcPdzADszmAL03wzuADwd+HaSXRhc2f9GkhUMgoHuFfvzqmqXtv1rk8zv5H2yqnYE9gCOTfKolr4IeGKr7zpgc+DF3U5V1bUMphT9+WR2oqruqKqvVNXhDIKHF41S5uSqWlhVC2dtssWalUiSJM1wBgCCwR2AvYFbqmp1Vd0CPJRBEHABg5P1I6tqbvtsA2yb5LHdSqrqauAo4C3DDVTVBQzm+/9te9j4JQwe2J1bVXOBfRj92YJ3A4dNtANJFiTZpi0/CNgF+MGk9l6SJKlHDAAEsIzB23/+eyhtVVXdxOCK/+eGtvlcSx/2IQZv9RntFZzvAf4CeCGwsqpWdvK+Bfx+ku5DwlTV5cDFk9iHrYEvJVnO4DmDO4ETJ7GdJElSr6SqproP0pSYPWdezTno+Knuhh4gfhGYJGk6S7Kkqhaui7r8JmD11s7bbsFiTwolSVLPOAVIkiRJ6hEDAEmSJKlHDAAkSZKkHjEAkCRJknrEAECSJEnqEQMASZIkqUcMACRJkqQeMQCQJEmSesQAQJIkSeoRAwBJkiSpRzaY6g5IU2XZylXMPeKsqe6G1iMrjn7hVHdBkqT7nXcAeijJuUmeP5T2xiQfbMuHJvl1ki06+XsmqSR/0kk7M8menTqvSnJZkiuTnJjkoUNt7NvqeGInbW6S5aP08ZQk+7XlvZNckuTSJFckeXVLP7LV94TOdoe2tIX3cZgkSZJmJAOAfjodOGAo7YCWDrAIuAjYd6jMDcDbx6n3wKraBdgFuAP4wlD+IuD8UdoeU5IHAycDf1JVTwZ2Bc7tFFk2VN9+wBWTrV+SJKlvDAD66dPA3klmw+AqPLANcH6SxwObAu9gcMLedSmwKslzx6u8qn4DvBl4TJIntzY2BXYDXsVaBADAZgymqt3c6r6jqq7q5H8e2Ke1sT2wCrhxLeqXJEnqFQOAHqqqm4ELgb1a0gHAJ6uqGJz0nw6cB+yQZOuhzd/FIDiYqI3VDAKGkek+LwK+WlVXA7ckWTDJvt4CfBH4QZLTkxyYpHvc/gL4nyQ7tb5/cjL1SpIk9ZUBQH91pwF1p/8cAJxRVXcBnwVe0t2oqs4DSLLHJNpIZ3kRcEZbPoM17y6Mqar+Eng2g6DlMOAjQ0XOaP1+EfC5cTuUHJJkcZLFq29bNdkuSJIkzRi+Bai/Pg/8c7sSv3FVXZxkF2Ae8I0kABsC3wc+MLTtuxk8C3DnWJUnmQXsDHwvycOBZwE7JSlgFlBJ3jzZzlbVMmBZko8D1wMHd7K/BLwPWFxVv2h9H6uekxk8U8DsOfNqsu1LkiTNFN4B6KmqupXBw7Qf4Z4P/x5ZVXPbZxtg2ySPHdr268CWwJNHq7s9uHsU8D9VdRmDB3NPrarHtnofzeAkfveJ+plk05E3DTXzgR8M9ed24C0MAhNJkiSNwwCg305ncBI/MjXnANacQvM5Rn9o993AdkNppyW5DFgOPIT2cC6DwGK43s8AL23LOyS5ofPpTjsK8Ob2itGlwD9wz6v/AFTVGVV18ei7KUmSpBEZPPcp9c/sOfNqzkHHT3U3tB7xi8AkSeurJEuqap18z5HPAKi3dt52CxZ7widJknrGKUCSJElSjxgASJIkST1iACBJkiT1iAGAJEmS1CMGAJIkSVKPGABIkiRJPWIAIEmSJPWIAYAkSZLUIwYAkiRJUo8YAEiSJEk9ssFUd0CaKstWrmLuEWdNdTc0w604+oVT3QVJku7BOwDroSSrkyxNcnmSS5O8KcmDWt6eSVYluSTJlUmO6Wx3ZJLDhupakWSrtvzIJJ9I8v0kS5JckGTfofJLkmzYtvtMJ32/JKcMlf1CkguG0o5MsrL1/4oki4byD2v9Xt727RUtfcMkxye5Lsk1re7tkjy81bU0yU86dS9t29zaqfv3knw5ybVJvpfkU0keeS9/DZIkSTOSAcD66faqml9VOwLPBf4Y+PtO/nlVtSuwK7B3kt0mqjBJgM8D36qq7avqKcABwHadMnOBlVX1m5a0MMmOY9T3UGAB8NAkjxvKPq6q5gP7AP+a5MFtm9e0/XlqVe0EPANI2+afgM2A36uqea2vnwVuaWMxH/jQSN3tM9JPkmwEnAWcVFVPqKonAScBj5hobCRJkvrEAGA9V1U/Aw4B3tBO4rt5twNLgW0nUdWzgN9U1Yc62/+gqv6lU+YFwFc768cAbxujvhcDXwLOYBBIjNb3a4DbgC1b0tuA11XVL1r+qqr6WJJNgL8ADq2q1S3vo8Adrd+T8VLggqr6Uqf9c6pq+SS3lyRJ6gUDgGmgqr7P4He1dTc9yZbAPOBbk6hmR+DiCcrsxT0DgE8BC5I8YZSyi4DT22fRKPkkWQBcU1U/S7IZsFlVXTdK0ScAPxwJDDoWt35Pxk7AkkmWlSRJ6i0DgOmje/V/jySXAT8Bzqyqn7T0GmPbNdKTfKDNwb+orW8IbNeCjRGrgfcBbx3a9pEMTtrPr6qrgTuT7NQpcmiSq4DvAkd2+j9W/8bKG2+beyXJIUkWJ1m8+rZV67JqSZKkacEAYBpIsj2Dk/GftaTzqmoXYGfgtUnmt/SbuXu6zYjNgJ8DlzOYsw9AVb0eeDZ3z5HfAzh/lOY/zmCu/mM6afu3dq5PsgKYyz2nAR1XVTu0cqcm2ahd3f9V25dh1wKPbXcJuhYAV4xSfjSXA0+ZqFBVnVxVC6tq4axNtphk1ZIkSTOHAcB6LskjGDz8emJV3eNqeLv6fhTwlpb0LeBPR06kk/wZcGmbV382sFGS13aq2KSzvBfwleH2q+q3wHHAGzvJi4C9qmpuVc1lcOK9xnMAVfVZBtN4DmpJRwEfSLJ569/mSQ6pql8BHwP+OcmslveK1r+zxxmerk8AT0/yu3cuJtkryc6T3F6SJKkXDADWTxuPvAYU+C/g68A/jFH2Q8Azkjyuqi4DTgTOT7IUeA3wlwAteHgR8EdJrk9yIYOT7pHgYU/g/43Rxr/TvjOivSnoMcB/j2RW1fXAL5L8wSjbvhMYeY3pScA5wEVJlrf2bmvl3gr8Grg6yTXAS4B9h4OesbQHovcG/rq9RvQK4GDuvmsiSZIkIJM8v9IMlmQ74MNV9YKp7ssDafaceTXnoOOnuhua4fwiMEnSupBkSVUtXCd1GQCorxYuXFiLFy+e6m5IkiRNaF0GAE4BkiRJknrEAECSJEnqEQMASZIkqUcMACRJkqQeMQCQJEmSesQAQJIkSeoRAwBJkiSpRwwAJEmSpB4xAJAkSZJ6ZIOp7oA0VZatXMXcI86a6m6oZ1Yc/cKp7oIkqee8AyBJkiT1iAHANJPk3CTPH0p7Y5IPJpmb5PYkSzufV7QyK5Js1dlmzyRntuWDk5zYyXtFkuVJLk9yRZLDWvopSfYbanvMNjtlPtfSr02yqlPu6W1/Fg7Vt7zTx1VDdT+n5a0eSn91Z/nWJFe15VPX1dhLkiTNBE4Bmn5OBw4AvtZJOwA4vC1fV1Xz723lSV4AvBF4XlX9KMlGwMsn2GzcNqtq31b3nsBhVbV3p72JunRet3zH7aO0+a+tznNbO4snqlySJKlvvAMw/Xwa2DvJbBhcMQe2Ac5fR/W/lcHJ848AqurXVfXhdVS3JEmSpph3AKaZqro5yYXAXsAXGFz9/2RVVbua/vgkSzub/HVVndeWz0myui1vClw5ShM7AUvWslvjtTkZpyW5vS1vCNzVydtjqO4XV9V1wMad9OtH7jJIkiRpfAYA09PINKCRAOCVnbzxpuM8s6pugrun46yj/tynaUfAgSPTddodjTM7eWszBWhCSQ4BDgGYtfkj1r6nkiRJ05xTgKanzwPPTrIA2LiqLl6HdV8OPGUd1rdeqaqTq2phVS2ctckWU90dSZKkB5wBwDRUVbcC5wIfYXA3YF06CnhvkkcBJJmd5G/WcRuSJEmaIgYA09fpwJOBM4bSHz/0esy1Onmvqi8DHwD+K8nlDJ4H6E4V+9ckN7TPBeuizQnsMVT3fhNvIkmSpLGkqqa6D9KUmD1nXs056Pip7oZ6xm8CliTdG0mWVNXCiUtOzIeA1Vs7b7sFiz0ZkyRJPeMUIEmSJKlHDAAkSZKkHjEAkCRJknrEAECSJEnqEQMASZIkqUcMACRJkqQeMQCQJEmSesQAQJIkSeoRAwBJkiSpR/wmYPXWspWrmHvEWVPdDUnACr+VW5IeMN4BkCRJknrEAEBrLcm+SSrJE9v6g5K8P8nyJMuSXJTkcS3vlS3tspa/T6eeDZLclOSoofpXJNmqs75nkjPb8sFJThwqvzjJ0iQ/THJjW16a5NH35zhIkiRNR04B0r2xCDgfOAA4Etgf2AbYparuSrId8Kv28+3AgqpalWRT4BGdep4HXAX8eZK3VVXdm85U1UKAJH8J7FRVb7yX+yVJkjTjeQdAa6WdxO8GvIpBAAAwB/hxVd0FUFU3VNX/AlsDvwRubem3VtX1neoWAScAPwSe9sDsgSRJUr8ZAGhtvQj4alVdDdySZAHwKeBP2rSbY5Ps2speCvwUuD7JR5P8yUglSTYGng2cCZzOIBiQJEnS/cwAQGtrEXBGWz4DWFRVNwA7AG8F7gK+meTZVbUa2AvYD7gaOC7JkW3bvYFzquo24DPAvklmtbzRpgLdq+lBw5Ic0p4ZWLz6tlXrokpJkqRpxWcANGlJHg48C9gpSQGzgEry5qq6A/gK8JUkP2Vwp+CbbV7/hcCFSb4BfJTBcwOLgN2SrGjVPxx4JvBfwM3AlsBNLe9hneX7pKpOBk4GmD1n3joJKiRJkqYT7wBobewHnFpVj62quVX1aOB64BlJtoHBG4GAXYAfJNmmTREaMb+lbw7sDjym1TMXeD13TwM6F3h5q28W8DLgnPt97yRJknrAOwBaG4uAo4fSPgOcwuB5gNkt7ULgROCRwDEtOPg1cCPwGuDPgLPbXYMRXwDe2+r4R+CkJJcCAb4K/Een7MFJXtRZf1qbhiRJkqQJ5F6+eVGa9mbPmVdzDjp+qrshCb8JWJImkmTJyKvP7yvvAKi3dt52CxZ70iFJknrGZwAkSZKkHjEAkCRJknrEAECSJEnqEQMASZIkqUcMACRJkqQeMQCQJEmSesQAQJIkSeoRAwBJkiSpRwwAJEmSpB7xm4DVW8tWrmLuEWdNdTckzRAr/GZxSdOEdwAkSZKkHjEAmMaSVJJjO+uHJTmys35Ikivb58Iku3fyzk2yuLO+MMm547R1WKtneZJLk7yiU89VLe2iJPM726xIslWnrx/v5G2Q5MYkZ7b1g5Oc2JaPTHJbkq075W8d6s++rc4ndtLmJlk+2fGTJEnqIwOA6e0O4M9GTrK7kuwNvBrYvaqeCLwG+ESSR3WKbZ3kBRM1kuQ1wHOBp1bVTsAzgHSKHFhVTwY+CLxvjGp+BeyUZOO2/lxg5TjN3gT833HyFwHnAwdM1H9JkiTdzQBgersTOBk4dJS8twCHV9VNAFV1MfAx4PWdMu8D3jGJdt4GvK6qftHqWlVVHxul3AXAtuPU8xVgZJLsIuD0ccp+BNg/ycOGM5JsCuwGvAoDAEmSpLViADD9fQA4MMkWQ+k7AkuG0ha39BEXAHckeeZYlSfZDNisqq6bRF/2Aj4/Tv4ZwAFJNgJ2Ab47TtlbGQQBfztK3ouAr1bV1cAtSRZMom+SJEnCAGDaa1flTwX+ZhLFA9RQ2rsY/y7AaNsMOy3JDQzuOvzLWIWq6jJgLoOr/1+eqLPA+4GDkmw+lL6IQTBB+7loEnUBv3suYnGSxatvWzXZzSRJkmYMA4CZ4XgG02Ee0km7AnjKULkFLf13qupsYCPgaSNpST6aZGmSL7cA41dJth+n/QOBxwGfYHBHYjxfBI5h/Ok/I337eavzdZ2+PRx4FvBvSVYAhzOYKpRRK1mzzpOramFVLZy1yfBNE0mSpJnPAGAGqKpbgE8xCAJGvBd4Tzthpr2d52AGD+oOezfw5k59f1FV86vqj1vSUcAHRq7EJ9k8ySFDffgtgzsJT0vypHG6+xHgnVW1bJK7988MHmYe+c6K/YBTq+qxVTW3qh4NXA/sPlYFkiRJupsBwMxxLPC7twFV1RcZnGx/J8mVwIeBl1XVj4c3rKovAzeOU/dJwDnARe01m/8PuG2Uem5v/ThsrIqq6oaqOmFSezQofxPwOWB2S1rU1rs+A7y0Le+Q5IbO5yWTbUuSJKkPUjXR9G5pZpo9Z17NOej4qe6GpBnCbwKWdH9KsqSqFq6LujaYuIg0M+287RYs9j9sSZLUM04BkiRJknrEAECSJEnqEQMASZIkqUcMACRJkqQeMQCQJEmSesQAQJIkSeoRAwBJkiSpRwwAJEmSpB4xAJAkSZJ6xG8CVm8tW7mKuUecNdXdkCTdByv8RndprXkHQJIkSeoRA4AplmS7JF9Ick2S65KckGTDJM9PsrR9bk1yVVs+NcmeSc4cqueUJPu15XM75Zcm+XRLPzLJypZ2RZJF4/TrkCRXts+FSXbv5A3XP9Lu6k7a0iRzW19XDaU/p5WvJMd26j0syZFteYfWztIk30tycksfr763J7k8yWUt/Q/W0a9JkiRpxnAK0BRKEuCzwElVtU+SWcDJwLur6nDga63cucBhVbW4re85ieoPHCk/5LiqOibJPGBJkk9X1W+H+rU38Gpg96q6KckC4PNJnlpVPxmn/turav5QXXOB86pq71H6cgfwZ0mOqqqbhvLe3/r6hVbPzp28NepL8ofA3sCCqrojyVbAhqO0KUmS1GveAZhazwJ+XVUfBaiq1cChwCuTbHJ/NlxV1wC3AVuOkv0W4PCRk/Kquhj4GPD6ddyNOxkEPIeOkjcHuKHT32UT1DUHuKmq7mjlb6qqH62rjkqSJM0UBgBTa0dgSTehqn4B/BB4wgTb7tGdBgP86VD+aZ389w1v3K7qX1NVP5tMv4DFLX20+h/e0jbupH1urL4meXwn7wPAgUm2GGrvOODsJF9JcmiSh05Q39eBRye5OskHk/zRKPs1MrVpcZLFq29bNVoRSZKkGc0pQFMrQK1Fetc9psEkOWUof6wpQIcm+Stge2Cv+9DXSU0BGq2vXVX1iySnAn8D3N5J/2iSr7U+7gO8OsmTx6svyVOAPYBnAp9MckRVnTLU3skM7jowe868icZYkiRpxvEOwNS6HFjYTUiyOfBo4Lr7qc3jqmoHYH/g1CQbjVLmCuApQ2kLWvr94XjgVcBDuolV9aOq+khV7cNgutBO41VSVaur6tyq+nvgDcCL76f+SpIkTVsGAFPrm8AmSV4B0B4CPhY4papuuz8brqrPMpjWc9Ao2e8F3jMytSfJfOBg4IP3U19uAT7FIAigtblXkge35UcBDwdWjlVHe2vQvE7SfOAH90d/JUmSpjOnAE2hqqok+wIfTPJ3DAKyLwNvWwfVn5ZkZErNTVX1nFHKvBP4RJIPV9VdnX59Mcm2wHeSFPBL4GVV9eN72Zc92nMKI95VVZ8eKnMsg6v2I54HnJDk12398Kr6SZInjlYfcD3wL+1ZgTuBa4FD7mV/JUmSZqxUOQ1a/TR7zryac9DxU90NSdJ94DcBqy+SLKmqhROXnJh3ANRbO2+7BYv9j0OSJPWMzwBIkiRJPWIAIEmSJPWIAYAkSZLUIwYAkiRJUo8YAEiSJEk9YgAgSZIk9YgBgCRJktQjBgCSJElSjxgASJIkST3iNwGrt5atXMXcI86a6m5IkqRpbMXRL5zqLqw17wBIkiRJPWIAMA0kuXWcvEuTnD6UdkqS65MsbfnP7uQ9OMnRSa5JsjzJhUle0PJWJFnWtlua5P2d+lYmmd3Wt2plN0pyZZKdO/W/OcmHOutfTXJOq+/aJKs69T89yblJFnbKz02yvC3vOVR+aZLntLzVQ+mv7izfmuSqtnzqfR1/SZKkmcQpQNNYkicxCOKekeQhVfWrTvbhVfXpJM8ETgbmtfR/BOYAO1XVHUkeCfxRZ7tnVtVNozS3GnglcNJIQlX9OskbgQ8meQawDfBqYGHr38bAw6rqqW19T+Cwqtq7sw8T7eZ53fIdt1fV/KG0f211ntvaWTxR5ZIkSX3jHYDp7aXAx4GvA386RpkLgG0BkmwC/BXw11V1B0BV/bSqPjWJto4HDk1yj6Cxqr4K/Bh4BXAccGRV/W/L3hM4dy32R5IkSfcz7wBMb/sDzwV2AN4AnD5Kmb2Az7flJwA/rKpfjFPnOUlWt+WPVdVxbfmHwPnAy4EvDW3zRuBC4Jqq+ngn/QWdtsdzWpLb2/KGwF2dvD2SLO2sv7iqrgM27qRfX1X7TqIdkhwCHAIwa/NHTGYTSZKkGcUAYJpK8n+AG6vqB0luAD6SZMvO1ff3JXkvsDXwtLWoeqwpQAD/BHwRuMerc6rqR0nOBs4cKr8bcNgk2jxwZLpOkrlD9azNFKAJVdXJDKZEMXvOvFrb7SVJkqY7pwBNX4uAJyZZAVwHbA68uJN/OIMr/u8APtbSrgUek2Sze9NgVV0LLAX+fJTsu+hcuU+yPfA/VfWbe9OWJEmS7h8GANNQkgcBLwF2qaq5VTUX2IdBUPA7VXUXcALwoCTPr6rbgH8H3p9kw1bXnCQvW4vm383kruq/APjqWtQrSZKkB4ABwPSwSZIbRj4M5tyvrKqVnTLfAn4/yZzuhlVVwLuAN7ekdwA3Ale0121+vq2PGHll56iv0Kyqy4GLJ9HnvVg3AcAeQ6/73G8d1ClJktRbGZwfSutO+76Ab1fVwgkLT6HZc+bVnIOOn+puSJKkaeyB+ibgJEvW1bmVDwFrnWuvGF2vT/4Bdt52CxZPw6/vliRJui+cAiRJkiT1iAGAJEmS1CMGAJIkSVKPGABIkiRJPeJbgNRbSX4JXDXV/VjPbAWM9U3QfeWYrMkxWZNjsibHZE2OyZock3sabzweW1WPWBeN+BYg9dlV6/urSh9oSRY7JvfkmKzJMVmTY7Imx2RNjsmaHJN7eqDGwylAkiRJUo8YAEiSJEk9YgCgPjt5qjuwHnJM1uSYrMkxWZNjsibHZE2OyZock3t6QMbDh4AlSZKkHvEOgCRJktQjBgDqpSR7JbkqybVJjpjq/qxrSVYkWZZkaZLFLe1hSb6R5Jr2c8uWniTvb2NxWZIFnXoOauWvSXJQJ/0prf5r27Z54PdyYkk+kuRnSZZ30u73cRirjak2xngcmWRlO1aWJvnjTt5b275dleT5nfRR/36SPC7Jd9t+fzLJhi19dlu/tuXPfWD2eGJJHp3knCTfS3J5kr9t6X0+TsYak94eK0k2SnJhkkvbmPxDS1/r/VhXYzXVxhmTU5Jc3zlO5rf0Gf+3A5BkVpJLkpzZ1tfPY6Sq/Pjp1QeYBVwHbA9sCFwK/P5U92sd7+MKYKuhtPcCR7TlI4D3tOU/Br4CBHga8N2W/jDg++3nlm15y5Z3IfCHbZuvAC+Y6n0eYxyeASwAlj+Q4zBWG1P9GWM8jgQOG6Xs77e/jdnA49rfzKzx/n6ATwEHtOUPAa9ty68DPtSWDwA+OdVj0dnPOcCCtrwZcHXb9z4fJ2ONSW+Plfa727QtPxj4bvv9r9V+rMuxmurPOGNyCrDfKOVn/N9O68+bgE8AZ96bY/2BOka8A6A+eipwbVV9v6p+A5wB7DPFfXog7AN8rC1/DHhRJ/3UGvhv4KFJ5gDPB75RVbdU1f8C3wD2anmbV9UFNfjX5tROXeuVqvoWcMtQ8gMxDmO1MaXGGI+x7AOcUVV3VNX1wLUM/nZG/ftpV+aeBXy6bT88tiPj8Wng2SNX8qZaVf24qi5uy78EvgdsS7+Pk7HGZCwz/lhpv+9b2+qD26dY+/1Yl2M1pcYZk7HM+L+dJNsBLwT+ra3fm2P9ATlGDADUR9sC/9NZv4Hx/3Objgr4epIlSQ5paY+sqh/D4D94YOuWPtZ4jJd+wyjp08UDMQ5jtbG+ekO7Jf+Rzq30tR2PhwM/r6o7h9LvUVfLX9XKr1faLfhdGVzJ9DhhjTGBHh8rbWrHUuBnDE5Sr2Pt92NdjtWUGx6Tqho5Tt7djpPjksxuaX342zkeeDNwV1u/N8f6A3KMGACoj0a7mjTTXoe1W1UtAF4AvD7JM8YpO9Z4rG36dNfXcTgJeDwwH/gxcGxLX5fjsd6PVZJNgc8Ab6yqX4xXdJS0GXmcjDImvT5Wqmp1Vc0HtmNwNfZJoxVrP9fVmKy34wFrjkmSnYC3Ak8E/g+DaT1vacVn1L4PS7I38LOqWtJNHqXoenGMGACoj24AHt1Z3w740RT15X5RVT9qP38GfI7Bf1Y/bbdUaT9/1oqPNR7jpW83Svp08UCMw1htrHeq6qftP/G7gA8zOFZg7cfjJga39DcYSr9HXS1/CyY/Fel+l+TBDE50T6uqz7bkXh8no42Jx8pAVf0cOJfBPPa13Y91OVbrjc6Y7NWmkFVV3QF8lHt/nEy3v53dgD9NsoLB9JxnMbgjsF4eIwYA6qOLgHntqfkNGTx888Up7tM6k+QhSTYbWQaeByxnsI8jb1c4CPhCW/4i8IoMPA1Y1W6pfg14XpIt263+5wFfa3m/TPK0NvfwFZ26poMHYhzGamO9M/KfaLMvg2MFBvtwQHtTxeOAeQweyBv176fN0T0H2K9tPzy2I+OxH3B2Kz/l2u/u34HvVdU/d7J6e5yMNSZ9PlaSPCLJQ9vyxsBzGDwbsbb7sS7HakqNMSZXdk7Mw2Auevc4mbF/O1X11qrarqrmMvj9nV1VB7K+HiO1Hjwx7cfPA/1h8DaCqxnM4Xz7VPdnHe/b9gzeDnApcPnI/jGYJ/hN4C9+v3kAAAD1SURBVJr282EtPcAH2lgsAxZ26nolgweQrgX+opO+kME/6tcBJ9K+VHB9+wCnM5iq8FsGV09e9UCMw1htTPVnjPH4eNvfy9p/PHM65d/e9u0qOm96Guvvpx17F7Zx+k9gdkvfqK1f2/K3n+qx6PR5dwa3yy8DlrbPH/f8OBlrTHp7rAC7AJe0fV8O/H/3dj/W1VhN9WecMTm7HSfLgf/g7jcFzfi/nU6/9+TutwCtl8eI3wQsSZIk9YhTgCRJkqQeMQCQJEmSesQAQJIkSeoRAwBJkiSpRwwAJEmSpB4xAJAkSZJ6xABAkiRJ6hEDAEmSJKlH/n9kzwFj4VpvdAAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f3c64d51be0>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "category_counts.toPandas().set_index(\"Category\").plot.barh(figsize = (10, 15))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+----------------------+\n",
      "|date                  |\n",
      "+----------------------+\n",
      "|04/20/2005 12:00:00 AM|\n",
      "|01/13/2008 12:00:00 AM|\n",
      "|05/05/2013 12:00:00 AM|\n",
      "|07/08/2003 12:00:00 AM|\n",
      "|10/04/2013 12:00:00 AM|\n",
      "|08/14/2007 12:00:00 AM|\n",
      "|03/04/2008 12:00:00 AM|\n",
      "|07/05/2006 12:00:00 AM|\n",
      "|12/10/2003 12:00:00 AM|\n",
      "|01/17/2011 12:00:00 AM|\n",
      "+----------------------+\n",
      "only showing top 10 rows\n",
      "\n"
     ]
    }
   ],
   "source": [
    "sfpd.select(\"date\").show(10, False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "datetime.date(2005, 4, 20)"
      ]
     },
     "execution_count": 53,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from datetime import datetime\n",
    "s = \"04/20/2005 12:00:00 AM\"\n",
    "d = datetime.strptime(s[:10], \"%m/%d/%Y\").date()\n",
    "d"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "datetime.date(2005, 4, 20)"
      ]
     },
     "execution_count": 54,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from datetime import datetime\n",
    "def parse_date(s):\n",
    "    return datetime.strptime(s[:10], \"%m/%d/%Y\").date()\n",
    "\n",
    "parse_date(\"04/20/2005 12:00:00 AM\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<function __main__.parse_date>"
      ]
     },
     "execution_count": 55,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from pyspark.sql.types import DateType\n",
    "spark.udf.register(\"parse_date\", parse_date, DateType())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+----------------------+----------------+\n",
      "|date                  |parse_date(date)|\n",
      "+----------------------+----------------+\n",
      "|04/20/2005 12:00:00 AM|2005-04-20      |\n",
      "|01/13/2008 12:00:00 AM|2008-01-13      |\n",
      "|05/05/2013 12:00:00 AM|2013-05-05      |\n",
      "|07/08/2003 12:00:00 AM|2003-07-08      |\n",
      "|10/04/2013 12:00:00 AM|2013-10-04      |\n",
      "|08/14/2007 12:00:00 AM|2007-08-14      |\n",
      "|03/04/2008 12:00:00 AM|2008-03-04      |\n",
      "|07/05/2006 12:00:00 AM|2006-07-05      |\n",
      "|12/10/2003 12:00:00 AM|2003-12-10      |\n",
      "|01/17/2011 12:00:00 AM|2011-01-17      |\n",
      "+----------------------+----------------+\n",
      "only showing top 10 rows\n",
      "\n"
     ]
    }
   ],
   "source": [
    "sfpd.select(expr(\"date\"), expr(\"parse_date(`date`)\")).show(10, False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>IncidntNum</th>\n",
       "      <th>Category</th>\n",
       "      <th>Descript</th>\n",
       "      <th>DayOfWeek</th>\n",
       "      <th>date</th>\n",
       "      <th>Time</th>\n",
       "      <th>PdDistrict</th>\n",
       "      <th>Resolution</th>\n",
       "      <th>Address</th>\n",
       "      <th>X</th>\n",
       "      <th>Y</th>\n",
       "      <th>Location</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>50436712</td>\n",
       "      <td>ASSAULT</td>\n",
       "      <td>BATTERY</td>\n",
       "      <td>Wednesday</td>\n",
       "      <td>2005-04-20</td>\n",
       "      <td>04:00</td>\n",
       "      <td>MISSION</td>\n",
       "      <td>NONE</td>\n",
       "      <td>18TH ST / CASTRO ST</td>\n",
       "      <td>-122.435003</td>\n",
       "      <td>37.760888</td>\n",
       "      <td>(37.7608878061245, -122.435002864271)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>80049078</td>\n",
       "      <td>LARCENY/THEFT</td>\n",
       "      <td>GRAND THEFT FROM A BUILDING</td>\n",
       "      <td>Sunday</td>\n",
       "      <td>2008-01-13</td>\n",
       "      <td>18:00</td>\n",
       "      <td>PARK</td>\n",
       "      <td>NONE</td>\n",
       "      <td>1100 Block of CLAYTON ST</td>\n",
       "      <td>-122.446838</td>\n",
       "      <td>37.762255</td>\n",
       "      <td>(37.7622550270122, -122.446837820235)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>130366639</td>\n",
       "      <td>ASSAULT</td>\n",
       "      <td>AGGRAVATED ASSAULT WITH A KNIFE</td>\n",
       "      <td>Sunday</td>\n",
       "      <td>2013-05-05</td>\n",
       "      <td>04:10</td>\n",
       "      <td>INGLESIDE</td>\n",
       "      <td>ARREST, BOOKED</td>\n",
       "      <td>0 Block of SGTJOHNVYOUNG LN</td>\n",
       "      <td>-122.444707</td>\n",
       "      <td>37.724931</td>\n",
       "      <td>(37.7249307267936, -122.444707063455)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>30810835</td>\n",
       "      <td>DRIVING UNDER THE INFLUENCE</td>\n",
       "      <td>DRIVING WHILE UNDER THE INFLUENCE OF ALCOHOL</td>\n",
       "      <td>Tuesday</td>\n",
       "      <td>2003-07-08</td>\n",
       "      <td>01:00</td>\n",
       "      <td>SOUTHERN</td>\n",
       "      <td>ARREST, BOOKED</td>\n",
       "      <td>MASON ST / TURK ST</td>\n",
       "      <td>-122.408954</td>\n",
       "      <td>37.783288</td>\n",
       "      <td>(37.7832878735491, -122.408953598286)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>130839567</td>\n",
       "      <td>OTHER OFFENSES</td>\n",
       "      <td>TRAFFIC VIOLATION ARREST</td>\n",
       "      <td>Friday</td>\n",
       "      <td>2013-10-04</td>\n",
       "      <td>20:53</td>\n",
       "      <td>TENDERLOIN</td>\n",
       "      <td>ARREST, BOOKED</td>\n",
       "      <td>TURK ST / LEAVENWORTH ST</td>\n",
       "      <td>-122.414056</td>\n",
       "      <td>37.782793</td>\n",
       "      <td>(37.7827931071006, -122.414056291891)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>70838580</td>\n",
       "      <td>BURGLARY</td>\n",
       "      <td>BURGLARY OF APARTMENT HOUSE, UNLAWFUL ENTRY</td>\n",
       "      <td>Tuesday</td>\n",
       "      <td>2007-08-14</td>\n",
       "      <td>07:00</td>\n",
       "      <td>NORTHERN</td>\n",
       "      <td>NONE</td>\n",
       "      <td>3100 Block of FRANKLIN ST</td>\n",
       "      <td>-122.426731</td>\n",
       "      <td>37.803467</td>\n",
       "      <td>(37.8034674969672, -122.426730544229)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>80233102</td>\n",
       "      <td>DRUG/NARCOTIC</td>\n",
       "      <td>POSSESSION OF MARIJUANA</td>\n",
       "      <td>Tuesday</td>\n",
       "      <td>2008-03-04</td>\n",
       "      <td>14:23</td>\n",
       "      <td>INGLESIDE</td>\n",
       "      <td>ARREST, CITED</td>\n",
       "      <td>MISSION ST / PERSIA AV</td>\n",
       "      <td>-122.435977</td>\n",
       "      <td>37.723129</td>\n",
       "      <td>(37.7231288306727, -122.43597721703)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>60711805</td>\n",
       "      <td>OTHER OFFENSES</td>\n",
       "      <td>DRIVERS LICENSE, SUSPENDED OR REVOKED</td>\n",
       "      <td>Wednesday</td>\n",
       "      <td>2006-07-05</td>\n",
       "      <td>15:50</td>\n",
       "      <td>INGLESIDE</td>\n",
       "      <td>ARREST, CITED</td>\n",
       "      <td>2300 Block of SAN JOSE AV</td>\n",
       "      <td>-122.447241</td>\n",
       "      <td>37.720158</td>\n",
       "      <td>(37.7201577971255, -122.447241159611)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>40062593</td>\n",
       "      <td>LARCENY/THEFT</td>\n",
       "      <td>GRAND THEFT FROM A BUILDING</td>\n",
       "      <td>Wednesday</td>\n",
       "      <td>2003-12-10</td>\n",
       "      <td>09:30</td>\n",
       "      <td>INGLESIDE</td>\n",
       "      <td>NONE</td>\n",
       "      <td>0 Block of MOFFITT ST</td>\n",
       "      <td>-122.432788</td>\n",
       "      <td>37.737157</td>\n",
       "      <td>(37.7371566745272, -122.432787775164)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>110051822</td>\n",
       "      <td>NON-CRIMINAL</td>\n",
       "      <td>STAY AWAY OR COURT ORDER, NON-DV RELATED</td>\n",
       "      <td>Monday</td>\n",
       "      <td>2011-01-17</td>\n",
       "      <td>15:35</td>\n",
       "      <td>INGLESIDE</td>\n",
       "      <td>NONE</td>\n",
       "      <td>600 Block of CAMPBELL AV</td>\n",
       "      <td>-122.408761</td>\n",
       "      <td>37.715900</td>\n",
       "      <td>(37.7159000951041, -122.408761072232)</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   IncidntNum                     Category  \\\n",
       "0    50436712                      ASSAULT   \n",
       "1    80049078                LARCENY/THEFT   \n",
       "2   130366639                      ASSAULT   \n",
       "3    30810835  DRIVING UNDER THE INFLUENCE   \n",
       "4   130839567               OTHER OFFENSES   \n",
       "5    70838580                     BURGLARY   \n",
       "6    80233102                DRUG/NARCOTIC   \n",
       "7    60711805               OTHER OFFENSES   \n",
       "8    40062593                LARCENY/THEFT   \n",
       "9   110051822                 NON-CRIMINAL   \n",
       "\n",
       "                                       Descript  DayOfWeek        date   Time  \\\n",
       "0                                       BATTERY  Wednesday  2005-04-20  04:00   \n",
       "1                   GRAND THEFT FROM A BUILDING     Sunday  2008-01-13  18:00   \n",
       "2               AGGRAVATED ASSAULT WITH A KNIFE     Sunday  2013-05-05  04:10   \n",
       "3  DRIVING WHILE UNDER THE INFLUENCE OF ALCOHOL    Tuesday  2003-07-08  01:00   \n",
       "4                      TRAFFIC VIOLATION ARREST     Friday  2013-10-04  20:53   \n",
       "5   BURGLARY OF APARTMENT HOUSE, UNLAWFUL ENTRY    Tuesday  2007-08-14  07:00   \n",
       "6                       POSSESSION OF MARIJUANA    Tuesday  2008-03-04  14:23   \n",
       "7         DRIVERS LICENSE, SUSPENDED OR REVOKED  Wednesday  2006-07-05  15:50   \n",
       "8                   GRAND THEFT FROM A BUILDING  Wednesday  2003-12-10  09:30   \n",
       "9      STAY AWAY OR COURT ORDER, NON-DV RELATED     Monday  2011-01-17  15:35   \n",
       "\n",
       "   PdDistrict      Resolution                      Address           X  \\\n",
       "0     MISSION            NONE          18TH ST / CASTRO ST -122.435003   \n",
       "1        PARK            NONE     1100 Block of CLAYTON ST -122.446838   \n",
       "2   INGLESIDE  ARREST, BOOKED  0 Block of SGTJOHNVYOUNG LN -122.444707   \n",
       "3    SOUTHERN  ARREST, BOOKED           MASON ST / TURK ST -122.408954   \n",
       "4  TENDERLOIN  ARREST, BOOKED     TURK ST / LEAVENWORTH ST -122.414056   \n",
       "5    NORTHERN            NONE    3100 Block of FRANKLIN ST -122.426731   \n",
       "6   INGLESIDE   ARREST, CITED       MISSION ST / PERSIA AV -122.435977   \n",
       "7   INGLESIDE   ARREST, CITED    2300 Block of SAN JOSE AV -122.447241   \n",
       "8   INGLESIDE            NONE        0 Block of MOFFITT ST -122.432788   \n",
       "9   INGLESIDE            NONE     600 Block of CAMPBELL AV -122.408761   \n",
       "\n",
       "           Y                               Location  \n",
       "0  37.760888  (37.7608878061245, -122.435002864271)  \n",
       "1  37.762255  (37.7622550270122, -122.446837820235)  \n",
       "2  37.724931  (37.7249307267936, -122.444707063455)  \n",
       "3  37.783288  (37.7832878735491, -122.408953598286)  \n",
       "4  37.782793  (37.7827931071006, -122.414056291891)  \n",
       "5  37.803467  (37.8034674969672, -122.426730544229)  \n",
       "6  37.723129   (37.7231288306727, -122.43597721703)  \n",
       "7  37.720158  (37.7201577971255, -122.447241159611)  \n",
       "8  37.737157  (37.7371566745272, -122.432787775164)  \n",
       "9  37.715900  (37.7159000951041, -122.408761072232)  "
      ]
     },
     "execution_count": 57,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sfpd_clean = sfpd.withColumn(\"date\", expr(\"parse_date(date)\"))\n",
    "sfpd_clean.limit(10).toPandas()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Caveat\n",
    "\n",
    "In the above example you used an UDF written in python. Python UDF make run data frame operations run significant slowly compared to UDF's writted in Scala or Java. However, it is just safe to use the built-in UDF or invoking UDF's written in Scala or Java. For the above exercise, there is already an built-in UDF defined called to_timestamp. Find out the details in the Spark API doc."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "### Exercise 5: This exercise is to explore data partitioning for file. Data partitions are  different from RDD partitions. For this exercise use weblogs dataset.\n",
    "\n",
    "1. Create a hive table - weblgos using weblogs dataset. Follow the steps mention in this doc. http://blog.einext.com/hadoop/hive-table-using-regex-serde\n",
    "2. Create a dataframe in Spark that refers to Hive table weblogs.\n",
    "3. Find total number of rows. \n",
    "4. Parse the time column as date time\n",
    "5. Save the weblogs data with partitioned by year and month based on the time field that you parsed in step #4\n",
    "6. Reload the partitioned dataset and verify the number of record maches with the original.\n",
    "\n",
    "\n",
    "You can upload weblogs to the HDFS and create a hive table by running the following hive table create command. \n",
    "\n",
    "```\n",
    "\n",
    "CREATE EXTERNAL TABLE `weblogs`(\n",
    "  `host` string COMMENT 'Host', \n",
    "  `identity` string COMMENT 'User Identity', \n",
    "  `user` string COMMENT 'User identifier', \n",
    "  `time` string COMMENT 'Date time of access', \n",
    "  `request` string COMMENT 'Http request', \n",
    "  `status` string COMMENT 'Http status', \n",
    "  `size` string COMMENT 'Http response size', \n",
    "  `referrer` string COMMENT 'Referrer url', \n",
    "  `useragent` string COMMENT 'Web client agent')\n",
    "ROW FORMAT SERDE \n",
    "  'org.apache.hadoop.hive.serde2.RegexSerDe' \n",
    "WITH SERDEPROPERTIES ( \n",
    "  'input.regex'='(\\\\d{1,3}\\\\.\\\\d{1,3}\\\\.\\\\d{1,3}\\\\.\\\\d{1,3})? (\\\\S+) (\\\\S+) (\\\\[.+?\\\\]) \\\"(.*?)\\\" (\\\\d{3}) (\\\\S+) \\\"(.*?)\\\" \\\"(.*?)\\\"', \n",
    "  'output.format.string'='%1$s %2$s %3$s %4$s %5$s %6$s %7$s %8$s %9$s') \n",
    "STORED AS INPUTFORMAT \n",
    "  'org.apache.hadoop.mapred.TextInputFormat' \n",
    "OUTPUTFORMAT \n",
    "  'org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat'\n",
    "LOCATION\n",
    "  '/user/cloudera/weblogs';\n",
    "  \n",
    "```\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Create a dataframe in Spark that refers to Hive table weblogs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+--------+---------+-----------+\n",
      "|database|tableName|isTemporary|\n",
      "+--------+---------+-----------+\n",
      "| default|   movies|      false|\n",
      "| default|  weblogs|      false|\n",
      "|        |   movies|       true|\n",
      "|        |  ratings|       true|\n",
      "|        |   stocks|       true|\n",
      "+--------+---------+-----------+\n",
      "\n"
     ]
    }
   ],
   "source": [
    "sql(\"show tables\").show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>host</th>\n",
       "      <th>identity</th>\n",
       "      <th>user</th>\n",
       "      <th>time</th>\n",
       "      <th>request</th>\n",
       "      <th>status</th>\n",
       "      <th>size</th>\n",
       "      <th>referrer</th>\n",
       "      <th>useragent</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>66.249.67.3</td>\n",
       "      <td>-</td>\n",
       "      <td>-</td>\n",
       "      <td>[20/Jul/2009:20:12:22 -0700]</td>\n",
       "      <td>GET /gallery/main.php?g2_controller=exif.Switc...</td>\n",
       "      <td>302</td>\n",
       "      <td>5</td>\n",
       "      <td>-</td>\n",
       "      <td>Mozilla/5.0 (compatible; Googlebot/2.1; +http:...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>66.249.67.3</td>\n",
       "      <td>-</td>\n",
       "      <td>-</td>\n",
       "      <td>[20/Jul/2009:20:12:25 -0700]</td>\n",
       "      <td>GET /gallery/main.php?g2_itemId=15741&amp;g2_fromN...</td>\n",
       "      <td>200</td>\n",
       "      <td>8068</td>\n",
       "      <td>-</td>\n",
       "      <td>Mozilla/5.0 (compatible; Googlebot/2.1; +http:...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>64.233.172.17</td>\n",
       "      <td>-</td>\n",
       "      <td>-</td>\n",
       "      <td>[20/Jul/2009:20:12:26 -0700]</td>\n",
       "      <td>GET /gwidgets/alexa.xml HTTP/1.1</td>\n",
       "      <td>200</td>\n",
       "      <td>2969</td>\n",
       "      <td>-</td>\n",
       "      <td>Mozilla/5.0 (compatible) Feedfetcher-Google; (...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>74.125.74.193</td>\n",
       "      <td>-</td>\n",
       "      <td>-</td>\n",
       "      <td>[20/Jul/2009:20:13:01 -0700]</td>\n",
       "      <td>GET /gwidgets/alexa.xml HTTP/1.1</td>\n",
       "      <td>200</td>\n",
       "      <td>2969</td>\n",
       "      <td>-</td>\n",
       "      <td>Mozilla/5.0 (compatible) Feedfetcher-Google; (...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>192.168.1.198</td>\n",
       "      <td>-</td>\n",
       "      <td>-</td>\n",
       "      <td>[20/Jul/2009:20:13:18 -0700]</td>\n",
       "      <td>GET / HTTP/1.1</td>\n",
       "      <td>200</td>\n",
       "      <td>17935</td>\n",
       "      <td>-</td>\n",
       "      <td>Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_5...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>192.168.1.198</td>\n",
       "      <td>-</td>\n",
       "      <td>-</td>\n",
       "      <td>[20/Jul/2009:20:13:18 -0700]</td>\n",
       "      <td>GET /style.css HTTP/1.1</td>\n",
       "      <td>200</td>\n",
       "      <td>1504</td>\n",
       "      <td>http://example.org/</td>\n",
       "      <td>Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_5...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>192.168.1.198</td>\n",
       "      <td>-</td>\n",
       "      <td>-</td>\n",
       "      <td>[20/Jul/2009:20:13:19 -0700]</td>\n",
       "      <td>GET /favicon.ico HTTP/1.1</td>\n",
       "      <td>404</td>\n",
       "      <td>146</td>\n",
       "      <td>http://example.org/</td>\n",
       "      <td>Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_5...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>66.249.67.3</td>\n",
       "      <td>-</td>\n",
       "      <td>-</td>\n",
       "      <td>[20/Jul/2009:20:13:21 -0700]</td>\n",
       "      <td>GET /gallery/main.php?g2_controller=exif.Switc...</td>\n",
       "      <td>302</td>\n",
       "      <td>5</td>\n",
       "      <td>-</td>\n",
       "      <td>Mozilla/5.0 (compatible; Googlebot/2.1; +http:...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>66.249.67.3</td>\n",
       "      <td>-</td>\n",
       "      <td>-</td>\n",
       "      <td>[20/Jul/2009:20:13:24 -0700]</td>\n",
       "      <td>GET /gallery/main.php?g2_itemId=30893&amp;g2_fromN...</td>\n",
       "      <td>200</td>\n",
       "      <td>8196</td>\n",
       "      <td>-</td>\n",
       "      <td>Mozilla/5.0 (compatible; Googlebot/2.1; +http:...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>66.249.67.3</td>\n",
       "      <td>-</td>\n",
       "      <td>-</td>\n",
       "      <td>[20/Jul/2009:20:13:29 -0700]</td>\n",
       "      <td>GET /gallery/main.php?g2_view=search.SearchSca...</td>\n",
       "      <td>200</td>\n",
       "      <td>6360</td>\n",
       "      <td>-</td>\n",
       "      <td>Mozilla/5.0 (compatible; Googlebot/2.1; +http:...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "            host identity user                          time  \\\n",
       "0    66.249.67.3        -    -  [20/Jul/2009:20:12:22 -0700]   \n",
       "1    66.249.67.3        -    -  [20/Jul/2009:20:12:25 -0700]   \n",
       "2  64.233.172.17        -    -  [20/Jul/2009:20:12:26 -0700]   \n",
       "3  74.125.74.193        -    -  [20/Jul/2009:20:13:01 -0700]   \n",
       "4  192.168.1.198        -    -  [20/Jul/2009:20:13:18 -0700]   \n",
       "5  192.168.1.198        -    -  [20/Jul/2009:20:13:18 -0700]   \n",
       "6  192.168.1.198        -    -  [20/Jul/2009:20:13:19 -0700]   \n",
       "7    66.249.67.3        -    -  [20/Jul/2009:20:13:21 -0700]   \n",
       "8    66.249.67.3        -    -  [20/Jul/2009:20:13:24 -0700]   \n",
       "9    66.249.67.3        -    -  [20/Jul/2009:20:13:29 -0700]   \n",
       "\n",
       "                                             request status   size  \\\n",
       "0  GET /gallery/main.php?g2_controller=exif.Switc...    302      5   \n",
       "1  GET /gallery/main.php?g2_itemId=15741&g2_fromN...    200   8068   \n",
       "2                   GET /gwidgets/alexa.xml HTTP/1.1    200   2969   \n",
       "3                   GET /gwidgets/alexa.xml HTTP/1.1    200   2969   \n",
       "4                                     GET / HTTP/1.1    200  17935   \n",
       "5                            GET /style.css HTTP/1.1    200   1504   \n",
       "6                          GET /favicon.ico HTTP/1.1    404    146   \n",
       "7  GET /gallery/main.php?g2_controller=exif.Switc...    302      5   \n",
       "8  GET /gallery/main.php?g2_itemId=30893&g2_fromN...    200   8196   \n",
       "9  GET /gallery/main.php?g2_view=search.SearchSca...    200   6360   \n",
       "\n",
       "              referrer                                          useragent  \n",
       "0                    -  Mozilla/5.0 (compatible; Googlebot/2.1; +http:...  \n",
       "1                    -  Mozilla/5.0 (compatible; Googlebot/2.1; +http:...  \n",
       "2                    -  Mozilla/5.0 (compatible) Feedfetcher-Google; (...  \n",
       "3                    -  Mozilla/5.0 (compatible) Feedfetcher-Google; (...  \n",
       "4                    -  Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_5...  \n",
       "5  http://example.org/  Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_5...  \n",
       "6  http://example.org/  Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_5...  \n",
       "7                    -  Mozilla/5.0 (compatible; Googlebot/2.1; +http:...  \n",
       "8                    -  Mozilla/5.0 (compatible; Googlebot/2.1; +http:...  \n",
       "9                    -  Mozilla/5.0 (compatible; Googlebot/2.1; +http:...  "
      ]
     },
     "execution_count": 59,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "weblogs = spark.table(\"weblogs\")\n",
    "weblogs.limit(10).toPandas()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Find total number of rows."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "239303"
      ]
     },
     "execution_count": 60,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "weblogs.count()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Parse the time column as date time"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+----------------------------+-------------------+\n",
      "|time                        |time_clean         |\n",
      "+----------------------------+-------------------+\n",
      "|[20/Jul/2009:20:12:22 -0700]|2009-07-20 20:12:22|\n",
      "|[20/Jul/2009:20:12:25 -0700]|2009-07-20 20:12:25|\n",
      "|[20/Jul/2009:20:12:26 -0700]|2009-07-20 20:12:26|\n",
      "|[20/Jul/2009:20:13:01 -0700]|2009-07-20 20:13:01|\n",
      "|[20/Jul/2009:20:13:18 -0700]|2009-07-20 20:13:18|\n",
      "|[20/Jul/2009:20:13:18 -0700]|2009-07-20 20:13:18|\n",
      "|[20/Jul/2009:20:13:19 -0700]|2009-07-20 20:13:19|\n",
      "|[20/Jul/2009:20:13:21 -0700]|2009-07-20 20:13:21|\n",
      "|[20/Jul/2009:20:13:24 -0700]|2009-07-20 20:13:24|\n",
      "|[20/Jul/2009:20:13:29 -0700]|2009-07-20 20:13:29|\n",
      "+----------------------------+-------------------+\n",
      "only showing top 10 rows\n",
      "\n"
     ]
    }
   ],
   "source": [
    "weblogs_clean = weblogs.withColumn(\"time_clean\", expr(r\"from_unixtime(UNIX_TIMESTAMP(TIME, '[dd/MMM/yyyy:HH:mm:ss Z]'))\"))\n",
    "weblogs_clean.select(\"time\", \"time_clean\").show(10, False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Save the weblogs data with partitioned by year and month based on the time field that you parsed in previous step"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {},
   "outputs": [],
   "source": [
    "(weblogs_clean\n",
    ".withColumn(\"year\", expr(\"year(time_clean)\"))\n",
    ".withColumn(\"month\", expr(\"month(time_clean)\"))\n",
    ".write\n",
    ".mode(\"overwrite\")\n",
    ".partitionBy(\"year\", \"month\")\n",
    ".save(\"weblogs-partitioned\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Reload the partitioned dataset and verify the number of record maches with the original."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "root\n",
      " |-- host: string (nullable = true)\n",
      " |-- identity: string (nullable = true)\n",
      " |-- user: string (nullable = true)\n",
      " |-- time: string (nullable = true)\n",
      " |-- request: string (nullable = true)\n",
      " |-- status: string (nullable = true)\n",
      " |-- size: string (nullable = true)\n",
      " |-- referrer: string (nullable = true)\n",
      " |-- useragent: string (nullable = true)\n",
      " |-- time_clean: string (nullable = true)\n",
      " |-- year: integer (nullable = true)\n",
      " |-- month: integer (nullable = true)\n",
      "\n"
     ]
    }
   ],
   "source": [
    "weblogs_partitioned = spark.read.load(\"weblogs-partitioned\")\n",
    "weblogs_partitioned.printSchema()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "239303"
      ]
     },
     "execution_count": 64,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "weblogs_partitioned.count()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Exercise 6: Convert RDD to Dataframe\n",
    "\n",
    "\n",
    "There are a couple of ways to conver the RDD into a dataframe. \n",
    "\n",
    "A. Supply the schema defined using StructType object  \n",
    "B. Infer the schema"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {},
   "outputs": [],
   "source": [
    "from random import random"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[7.576205089743215e-05,\n",
       " 0.5286446509604893,\n",
       " 0.7090196027486819,\n",
       " 0.7064680098536666,\n",
       " 0.8138804138909966,\n",
       " 0.9456380937096995,\n",
       " 0.25481721212652253,\n",
       " 0.9486494217992113,\n",
       " 0.35200515066138993,\n",
       " 0.770356992832654]"
      ]
     },
     "execution_count": 66,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "rdd = sc.parallelize([random() for _ in range(10)])\n",
    "rdd.collect()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+--------------------+\n",
      "|                col1|\n",
      "+--------------------+\n",
      "|7.576205089743215E-5|\n",
      "|  0.5286446509604893|\n",
      "|  0.7090196027486819|\n",
      "|  0.7064680098536666|\n",
      "|  0.8138804138909966|\n",
      "|  0.9456380937096995|\n",
      "| 0.25481721212652253|\n",
      "|  0.9486494217992113|\n",
      "| 0.35200515066138993|\n",
      "|   0.770356992832654|\n",
      "+--------------------+\n",
      "\n"
     ]
    }
   ],
   "source": [
    "from pyspark.sql import Row\n",
    "rddRow = rdd.map(lambda f: Row(f))\n",
    "spark.createDataFrame(rddRow).toDF(\"col1\").show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+--------------------+\n",
      "|                col1|\n",
      "+--------------------+\n",
      "|7.576205089743215E-5|\n",
      "|  0.5286446509604893|\n",
      "|  0.7090196027486819|\n",
      "|  0.7064680098536666|\n",
      "|  0.8138804138909966|\n",
      "|  0.9456380937096995|\n",
      "| 0.25481721212652253|\n",
      "|  0.9486494217992113|\n",
      "| 0.35200515066138993|\n",
      "|   0.770356992832654|\n",
      "+--------------------+\n",
      "\n"
     ]
    }
   ],
   "source": [
    "rddRow = rdd.map(lambda f: Row(col1 = f))\n",
    "df = spark.createDataFrame(rddRow)\n",
    "df.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Supply schema while converting the rdd into a dataframe"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "By default, spark will try to infer the column names and a types by sampling the row object RDD. To control this process you can supply the schema programmaitcally as well.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "root\n",
      " |-- c1: double (nullable = true)\n",
      " |-- c2: string (nullable = true)\n",
      " |-- c3: null (nullable = true)\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# Schema created by schema inferencing\n",
    "rdd = sc.parallelize([\n",
    "    Row(c1 = 1.0, c2 = None, c3 = None), \n",
    "    Row(c1 = None, c2= \"Apple\", c3 = None)])\n",
    "df = spark.createDataFrame(rdd, samplingRatio=1.0) # samplingRatio = 1 forces to see all records\n",
    "df.printSchema()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, suppose you already know the schema of the record - it should be three columns. C1, C2 and C3 of float, string and date types respectively."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "metadata": {},
   "outputs": [],
   "source": [
    "from pyspark.sql.types import *"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "metadata": {},
   "outputs": [],
   "source": [
    "schema = StructType([\n",
    "    StructField(\"c1\", FloatType()),\n",
    "    StructField(\"c2\", StringType()),\n",
    "    StructField(\"c3\", DateType()),\n",
    "])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "root\n",
      " |-- c1: float (nullable = true)\n",
      " |-- c2: string (nullable = true)\n",
      " |-- c3: date (nullable = true)\n",
      "\n"
     ]
    }
   ],
   "source": [
    "df = spark.createDataFrame(rdd, schema)\n",
    "df.printSchema()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+----+-----+----+\n",
      "|  c1|   c2|  c3|\n",
      "+----+-----+----+\n",
      "| 1.0| null|null|\n",
      "|null|Apple|null|\n",
      "+----+-----+----+\n",
      "\n"
     ]
    }
   ],
   "source": [
    "df.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['movieId,title,genres',\n",
       " '1,Toy Story (1995),Adventure|Animation|Children|Comedy|Fantasy',\n",
       " '2,Jumanji (1995),Adventure|Children|Fantasy',\n",
       " '3,Grumpier Old Men (1995),Comedy|Romance',\n",
       " '4,Waiting to Exhale (1995),Comedy|Drama|Romance',\n",
       " '5,Father of the Bride Part II (1995),Comedy',\n",
       " '6,Heat (1995),Action|Crime|Thriller',\n",
       " '7,Sabrina (1995),Comedy|Romance',\n",
       " '8,Tom and Huck (1995),Adventure|Children',\n",
       " '9,Sudden Death (1995),Action']"
      ]
     },
     "execution_count": 74,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "moviesRdd = sc.textFile(\"movielens/movies.csv\")\n",
    "moviesRdd.take(10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['11,\"American President, The (1995)\",Comedy|Drama|Romance',\n",
       " '29,\"City of Lost Children, The (Cité des enfants perdus, La) (1995)\",Adventure|Drama|Fantasy|Mystery|Sci-Fi',\n",
       " '40,\"Cry, the Beloved Country (1995)\",Drama',\n",
       " '50,\"Usual Suspects, The (1995)\",Crime|Mystery|Thriller',\n",
       " '54,\"Big Green, The (1995)\",Children|Comedy',\n",
       " '58,\"Postman, The (Postino, Il) (1994)\",Comedy|Drama|Romance',\n",
       " '59,\"Confessional, The (Confessionnal, Le) (1995)\",Drama|Mystery',\n",
       " '60,\"Indian in the Cupboard, The (1995)\",Adventure|Children|Fantasy',\n",
       " '73,\"Misérables, Les (1995)\",Drama|War',\n",
       " '78,\"Crossing Guard, The (1995)\",Action|Crime|Drama|Thriller']"
      ]
     },
     "execution_count": 75,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "moviesRdd.filter(lambda line: len(line.split(\",\")) != 3).take(10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 76,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+-------+----------------------------------+-------------------------------------------+\n",
      "|movieId|title                             |genre                                      |\n",
      "+-------+----------------------------------+-------------------------------------------+\n",
      "|1      |Toy Story (1995)                  |Adventure|Animation|Children|Comedy|Fantasy|\n",
      "|2      |Jumanji (1995)                    |Adventure|Children|Fantasy                 |\n",
      "|3      |Grumpier Old Men (1995)           |Comedy|Romance                             |\n",
      "|4      |Waiting to Exhale (1995)          |Comedy|Drama|Romance                       |\n",
      "|5      |Father of the Bride Part II (1995)|Comedy                                     |\n",
      "|6      |Heat (1995)                       |Action|Crime|Thriller                      |\n",
      "|7      |Sabrina (1995)                    |Comedy|Romance                             |\n",
      "|8      |Tom and Huck (1995)               |Adventure|Children                         |\n",
      "|9      |Sudden Death (1995)               |Action                                     |\n",
      "|10     |GoldenEye (1995)                  |Action|Adventure|Thriller                  |\n",
      "+-------+----------------------------------+-------------------------------------------+\n",
      "only showing top 10 rows\n",
      "\n"
     ]
    }
   ],
   "source": [
    "from io import StringIO\n",
    "import csv\n",
    "from pyspark.sql.types import Row\n",
    "\n",
    "def parse_movie(line):\n",
    "    tokens = [v for v in csv.reader(StringIO(line), delimiter=',')][0]\n",
    "    fields = tuple(tokens)\n",
    "    return Row(*tokens)\n",
    "\n",
    "moviesCleanRdd = (moviesRdd\n",
    "                  .filter(lambda line: not line.startswith(\"movieId\"))\n",
    "                  .map(parse_movie)\n",
    "                 )\n",
    "df = moviesCleanRdd.toDF().toDF(\"movieId\", \"title\", \"genre\")\n",
    "df.show(10, False) "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 77,
   "metadata": {},
   "outputs": [],
   "source": [
    "schema = StructType([\n",
    "    StructField(\"movieId\", StringType()),\n",
    "    StructField(\"title\", StringType()),\n",
    "    StructField(\"genres\", StringType()),\n",
    "])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 78,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+-------+----------------------------------+-------------------------------------------+\n",
      "|movieId|title                             |genres                                     |\n",
      "+-------+----------------------------------+-------------------------------------------+\n",
      "|1      |Toy Story (1995)                  |Adventure|Animation|Children|Comedy|Fantasy|\n",
      "|2      |Jumanji (1995)                    |Adventure|Children|Fantasy                 |\n",
      "|3      |Grumpier Old Men (1995)           |Comedy|Romance                             |\n",
      "|4      |Waiting to Exhale (1995)          |Comedy|Drama|Romance                       |\n",
      "|5      |Father of the Bride Part II (1995)|Comedy                                     |\n",
      "|6      |Heat (1995)                       |Action|Crime|Thriller                      |\n",
      "|7      |Sabrina (1995)                    |Comedy|Romance                             |\n",
      "|8      |Tom and Huck (1995)               |Adventure|Children                         |\n",
      "|9      |Sudden Death (1995)               |Action                                     |\n",
      "|10     |GoldenEye (1995)                  |Action|Adventure|Thriller                  |\n",
      "+-------+----------------------------------+-------------------------------------------+\n",
      "only showing top 10 rows\n",
      "\n"
     ]
    }
   ],
   "source": [
    "spark.createDataFrame(moviesCleanRdd, schema).show(10, False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Exercise 7: Data format\n",
    "\n",
    "Save stocks.csv dataset in the following formats and compare the size on disk\n",
    "\n",
    " A. csv \n",
    " B. Json\n",
    " C. Parquet"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 79,
   "metadata": {},
   "outputs": [],
   "source": [
    "stocks = (spark\n",
    "          .read\n",
    "          .option(\"header\", True)\n",
    "          .option(\"inferSchema\", True)\n",
    "          .csv(\"stocks\"))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 80,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Found 1 items\r\n",
      "-rw-r--r--   1 cloudera cloudera    121.8 M 2018-04-22 23:12 stocks/stocks.csv\r\n"
     ]
    }
   ],
   "source": [
    "!hadoop fs -ls -h stocks"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 81,
   "metadata": {},
   "outputs": [],
   "source": [
    "(stocks\n",
    ".write\n",
    ".format(\"csv\")\n",
    ".option(\"compression\", \"gzip\")\n",
    ".option(\"header\", True)\n",
    ".save(\"stocks.csv.gz\"))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 82,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Found 3 items\r\n",
      "-rw-r--r--   1 cloudera cloudera          0 2018-04-24 03:38 stocks.csv.gz/_SUCCESS\r\n",
      "-rw-r--r--   1 cloudera cloudera     23.5 M 2018-04-24 03:38 stocks.csv.gz/part-00000-a9562463-54d1-46a3-9491-d28bc4346fde-c000.csv.gz\r\n",
      "-rw-r--r--   1 cloudera cloudera     21.8 M 2018-04-24 03:38 stocks.csv.gz/part-00001-a9562463-54d1-46a3-9491-d28bc4346fde-c000.csv.gz\r\n"
     ]
    }
   ],
   "source": [
    "!hadoop fs -ls -h stocks.csv.gz"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Save in json format"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 83,
   "metadata": {},
   "outputs": [],
   "source": [
    "stocks.write.format(\"json\").save(\"stocks.json\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 84,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Found 3 items\r\n",
      "-rw-r--r--   1 cloudera cloudera          0 2018-04-24 03:38 stocks.json/_SUCCESS\r\n",
      "-rw-r--r--   1 cloudera cloudera    145.6 M 2018-04-24 03:38 stocks.json/part-00000-f50b95eb-86da-4f83-ab3e-63510cc083cd-c000.json\r\n",
      "-rw-r--r--   1 cloudera cloudera    133.9 M 2018-04-24 03:38 stocks.json/part-00001-f50b95eb-86da-4f83-ab3e-63510cc083cd-c000.json\r\n"
     ]
    }
   ],
   "source": [
    "!hadoop fs -ls -h stocks.json"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Save in parquet format"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 85,
   "metadata": {},
   "outputs": [],
   "source": [
    "stocks.write.format(\"parquet\").save(\"stocks.parquet\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 86,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Found 3 items\r\n",
      "-rw-r--r--   1 cloudera cloudera          0 2018-04-24 03:39 stocks.parquet/_SUCCESS\r\n",
      "-rw-r--r--   1 cloudera cloudera     20.9 M 2018-04-24 03:39 stocks.parquet/part-00000-8ed1fcb0-65b0-4a25-9044-f6a1ffcf27a8-c000.snappy.parquet\r\n",
      "-rw-r--r--   1 cloudera cloudera     19.1 M 2018-04-24 03:39 stocks.parquet/part-00001-8ed1fcb0-65b0-4a25-9044-f6a1ffcf27a8-c000.snappy.parquet\r\n"
     ]
    }
   ],
   "source": [
    "!hadoop fs -ls -h stocks.parquet"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Observations\n",
    "\n",
    " - CSV file format: 121 MB\n",
    " - CSV gzip compressed: 45.3 MB\n",
    " - Json uncompressed: 279.5 MB\n",
    " - Parquet snappy compressed: 40 MB"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Exercise 8: Size of dataframe in cache\n",
    "\n",
    "Cache the stocks.csv in RDD format and do the same in Dataframe format and compare the memory utilization."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 87,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1857093"
      ]
     },
     "execution_count": 87,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "stocksRdd = sc.textFile(\"stocks\")\n",
    "stocksRdd.cache().count()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Check the storage table in Spark Web UI. You can find Spark Web UI as below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 88,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'http://quickstart.cloudera:4041'"
      ]
     },
     "execution_count": 88,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sc.uiWebUrl"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 89,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1857092"
      ]
     },
     "execution_count": 89,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "stocksDf = spark.read.option(\"header\", True).option(\"inferSchema\", True).csv(\"stocks\")\n",
    "stocksDf.cache()\n",
    "stocksDf.count()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
